September 28, 2015

Comparing an A/B Test to a Multivariate Test

In todays world many business thrive from online audiences and daily traffic. In order to have a site reach its full potential takes a lot time and optimizations. You’ve might of heard the term “A/B Test” or at times “Multivariate Test” (MVT) floating around on the web. There seems to be much confusion over these two common testing methods. I would like to clear up any misconceptions and provide a detailed description and explain why they are different.

We’ll take a look at their usefulness, advantages and disadvantages so that you can decide which type to implement in the future.

A/B Testing

So let’s start with the most common A/B Testing, this is also known as split testing to most. As the name states A/B testing is a simple concept in that version A (the control) is tested against another version B (the variation). Other versions C, D etc can be tested in the same A/B test because you are testing these against the original control variation. Visitors are split into each version of the site, if there are two versions running typically a 50% weight is given to each variation to perform an even test down the middle. The test has to be measurable, so optimizers will choose some type of Call To Action (CTA) to define which version is the winner. For example if your CTA is a visitor form fill, the version that receives more conversions/form fills would be the winner of the test, you would then implement the winner into your website changes.




Using this simplified method will allow you to generate results quickly with a solid conclusion. For sites that have a small amount of daily visitors, running this type of test is ideal.

  • Some companies only use A/B testing because of it’s powerful and quick results.
  • Test only take minutes to setup and hours to verify data depending on the amount of traffic to your site.


Simplicity is also the reason A/B testing has limitations, any optimizer needing to draw many conclusions at once through multiple variations would have to wait for each test to finish before starting another this would take an increased amount of time to reach the end goal.

  • A/B will not give you any deeper insights into other elements on the page.
  • Testing more than a few elements at once is not possible.


Multivariate Testing (MVT)

MVT is similar to A/B testing but instead of selecting one element at a time to test a variation of, multiple variation tests are running. More variations means more information about how these elements interact with eachother. If you ran a test on a landing page and wanted to test two main images and two headlines this would require four versions of the page to be created. This speeds up the process of having to run small A/B tests individually.



MVT is a powerful way to optimize a web page or landing page very quickly, it’s possible to test multiple headlines, images, text, and buttons at once.

  • MVT allows for multiple small test to be ran at once to allow incremental conversion rate improvement.
  • Smaller page elements can be compared quickly to see their effect on conversion rates.


There are some downsides to using MVT largely because you need a high volume of traffic to complete the tests.

  • MVT is not ideal for major layout changes
  • Can cause restrictions in marketing creativity.
  • Required many more variable combinations to be run than A/B testing.
  • Tests can become complicated very quickly if there are too many variations added at once time.

Final Words

Most optimizers will use A/B testing and its method of simplicity to test large radical changes such as a landing page design or a main elements. Multivariate testing is often used to refine and optimize an existing page. If you are a beginning optimizer starting with A/B testing may be best for it’s ease of use and reliable data. Rather than picking one method over the other, they can also be used in conjunction for best results and deeper insights. One last thing, “Always Be Testing” I’ve heard this phrase from some of the top CRO’s in the industry, the internet and it’s users are constantly changing, so never stop testing.