To A/B or Not To A/B? That Is The Question

If you're new to internet marketing or if you've been in the industry for years, you've inevitably read about, thought about or tinkered about with online testing. I myself have written about the need for testing in PPC (keyword match types, ad copy, landing pages) and the importance of recording & measuring test performance. I can actually trace my own fascination with online testing to Pubcon 2009 in Las Vegas. I attended a session on landing pages with Kate Morris, Tim Ash, Joanna Lord and Brad Geddes that gave me new ways to think about testing. I also learned about the concept of post-click marketing from the Janet Driscoll Miller, Anna Talerico, Scott Brinker, and Chris Goward. This was mind-blowing stuff to me at the time and I'm still working to master it.
However, you won't get very far in online testing without coming across two very different strategies: A/B testing and multivariate testing.

What Is A/B Testing

As the name implies, A/B testing has at least 2 distinct versions (there can be more) that are tested against each other. Most often you will have the best current performer (the control) being tested against 1 or more new versions.

What Is Multivariate Testing

Multivariate testing takes one version and then tests different elements on the page in varied combinations. For example, you could have 2 different headlines being tested, 3 different product images being tested, and 2 different buttons being tested. Code on the page rotates the versions of each element so you test each possible combination. In our example, there are 12 different "pages" (2x3x2) that any specific user could have seen during the test period.

Which Should You Use?

I know it sounds like a cop-out, but it depends. Here are my general guidelines for when you should use A/B testing and when you should use multivariate testing:

• You're just starting to do online testing - Use A/B. It's easier to implement and you can test drastically different versions to gain very broad knowledge about your customers.
• You want to start by testing a small amount of traffic - Use A/B. When testing you need to achieve statistical validity. This can be calculated easily, but basically you need enough data to make sure your conclusion isn't a result of random chance. It's easier to achieve validity with 2 or 3 versions than it is with the numerous versions in a multivariate test.
• You've done some testing and you're refining your tests - Use multivariate. The ability to test multiple elements in different combinations allows you to become much more granular with your analysis and your findings.
• You have lots of traffic - Use multivariate (but only if you're testing worthwhile elements). A/B tests can still get great results with high traffic pages, but you'll need to move fast with your iterations because validity will be achieved more quickly.

These guidelines should help you determine which testing approach is best for your situation and how you can get started.

P.S. Remember that with either method you need to test elements that will impact results. Changing the text of the privacy policy link in your footer is too insignificant. Changing the headline or main image can create dramatic results.

If you liked this, you might also enjoy Think - Test - Track : The Improvement Framework