Multivariate Testing on Websites
Multivariate testing (MVT) allows you to simultaneously test multiple changes on a page and identify the most effective combination. Unlike A/B testing, MVT reveals interactions between elements.
A/B vs Multivariate
| Criterion | A/B Test | MVT |
|---|---|---|
| Changes | 1 (2 variants) | 2+ elements × N variants |
| Traffic Volume | Lower | Significantly higher |
| Speed | Faster | Slower |
| Element Interaction | Does not detect | Detects |
| Goal | Single winner | Best combination |
Use MVT when: changes are interdependent, traffic is sufficient, and you need to understand interactions between headline + CTA + image.
Planning an MVT Experiment
Example: testing on a product page
- Headline: 2 variants (A, B)
- Image: 3 variants (A, B, C)
- CTA button: 2 variants (A, B)
Total: 2 × 3 × 2 = 12 combinations. Each combination is a separate variant.
Calculate Required Traffic
def mvt_sample_size(n_combinations, baseline_cr, mde=0.05, alpha=0.05, power=0.8):
"""Each combination requires the same traffic as an A/B test"""
from scipy import stats
import math
p1 = baseline_cr
p2 = baseline_cr * (1 + mde)
p_avg = (p1 + p2) / 2
z_a = stats.norm.ppf(1 - alpha / 2)
z_b = stats.norm.ppf(power)
n_per_combo = ((z_a * math.sqrt(2 * p_avg * (1-p_avg)) +
z_b * math.sqrt(p1*(1-p1) + p2*(1-p2))) / (p2-p1)) ** 2
total = n_per_combo * n_combinations
print(f"Per combination: {math.ceil(n_per_combo):,}")
print(f"Total needed: {math.ceil(total):,}")
print(f"At 1000 daily visitors: {math.ceil(total/1000)} days")
mvt_sample_size(n_combinations=12, baseline_cr=0.04, mde=0.15)
# Per combination: 4,519
# Total needed: 54,228
# At 1000 daily visitors: 55 days
If traffic is insufficient for full MVT — use fractional factorial design (test a subset of combinations).
Implementation via Optimizely
// Optimizely Snippet (add to <head>)
<script src="https://cdn.optimizely.com/js/PROJECT_ID.js"></script>
// Programmatic approach
const optimizely = window.optimizely || []
// Get variation for specific experiment
const variationKey = optimizely.get('state').getVariationMap()['mvt_homepage_elements']
// variationKey = "headline_B_image_C_cta_A"
// Implementation without Optimizely
function getMVTVariant(userId, elements) {
const variants = {}
for (const [element, options] of Object.entries(elements)) {
const hash = cyrb53(`${userId}_${element}`)
variants[element] = options[hash % options.length]
}
return variants
}
const elements = {
headline: ['Buy today', '20% off for new customers'],
image: ['lifestyle', 'product-white', 'product-action'],
cta: ['Add to cart', 'Buy now'],
}
const variants = getMVTVariant(userId, elements)
// variants = { headline: 'Buy today', image: 'product-action', cta: 'Buy now' }
// Apply variants to DOM
applyVariants(variants)
// Record in analytics
gtag('event', 'mvt_assignment', {
experiment: 'product_page_mvt',
headline: variants.headline,
image: variants.image,
cta: variants.cta,
combination: Object.values(variants).join('_')
})
MVT Results Analysis
import pandas as pd
from scipy import stats
def analyze_mvt(data):
"""data: DataFrame with columns combination, visitors, conversions"""
data['cvr'] = data['conversions'] / data['visitors']
data = data.sort_values('cvr', ascending=False)
# Find best combination
best = data.iloc[0]
control = data[data['combination'] == 'baseline'].iloc[0]
print(f"\nTop combinations:")
print(data.head(5).to_string())
# Statistical significance of best vs control
from scipy.stats import proportions_ztest
_, p_value = proportions_ztest(
[best['conversions'], control['conversions']],
[best['visitors'], control['visitors']]
)
print(f"\nBest: {best['combination']} CVR={best['cvr']:.2%}")
print(f"Control: CVR={control['cvr']:.2%}")
print(f"P-value: {p_value:.4f}")
# Main effects analysis
# How much each element individually affects conversion
for element in ['headline', 'image', 'cta']:
effect = data.groupby(element)['cvr'].mean()
print(f"\nMain effect of {element}:")
print(effect.sort_values(ascending=False))
Fractional Factorial Design
If you cannot test all 12 combinations:
# Use Latin Square or Taguchi method
# Test 4 combinations instead of 12, covering interactions
combinations_to_test = [
{'headline': 'A', 'image': 'A', 'cta': 'A'}, # control
{'headline': 'A', 'image': 'B', 'cta': 'B'},
{'headline': 'B', 'image': 'A', 'cta': 'B'},
{'headline': 'B', 'image': 'B', 'cta': 'A'},
]
# Reduces traffic by 3x but loses some interactions
Delivery Time
Planning and implementing MVT with main effects analysis — 5–7 business days.