Skip to main content

Documentation Index

Fetch the complete documentation index at: https://userstory.zeroemdashes.com/llms.txt

Use this file to discover all available pages before exploring further.

A/B testing gives you a clean answer to a clean question: is version A or version B better? Multivariate testing asks a messier question: which combination of changes produces the best result? 🧪

​
What it is

Multivariate testing (MVT) lets you test multiple variables simultaneously - say, three headline options and two button colours - and measure every combination at once. Instead of running separate A/B tests sequentially, you get data on all permutations in a single experiment. A test with 3 headlines and 2 button colours produces 6 combinations (3 x 2). Add another variable with 3 options and you’re at 18. The combinations multiply fast.

​
When to use it over A/B testing

MVT makes sense when:
  • You want to understand how variables interact - does headline A work better with button colour 1, or only with button colour 2?
  • You have high traffic and can afford to split it many ways without sacrificing statistical power
  • You’re optimising a mature page or flow where incremental gains matter
A/B testing is the right choice for most teams most of the time. MVT is a specialist tool for when interaction effects matter and you have the traffic to support it 💡

​
The traffic problem

This is the real constraint. Every combination needs its own sample to reach significance. If an A/B test needs 1,000 users per variant, an 18-combination MVT needs 18,000 - just for the same confidence level. Most products don’t have that kind of traffic. Running an underpowered MVT that never reaches significance is a waste of weeks. When in doubt, run sequential A/B tests instead.

​
What you actually learn

The unique value of MVT isn’t just picking the winning combination - it’s understanding interaction effects. You might discover that a bold headline works brilliantly with a subtle CTA but poorly with an aggressive one. That’s a learning you can’t get from running those variables in separate tests. Lesson learned: the teams I’ve seen get the most from MVT were running it on high-traffic landing pages with a dedicated experimentation function. If you’re a small team with average traffic, sequential A/B testing will serve you better and teach you just as much 🙌