Which Test Won deserves credit for evangelizing the value and best practices of A/B and Multivariate testing. Unfortunately, this never ending parade of winning results fabricates the impression that testing is all about giant golden nuggets. This creates false expectations followed by big disappointments.
Strategy: knowledge of and engagement with customers is the only competitive advantage in the age of the customer. Testing is an essential tool of such engagement. By not testing companies are in essence choosing to lose a competitive race.
With the right approach and with proper instrumentation every test can be a winner.
A more promising approach is to combine test and web analytics data.
On the surface this looks like the right approach. Web analytics solutions already track a broad spectrum of visitor attributes and custom goals. The integration between two is a fairly simple exercise.
Challenges: The integration requires custom tagging of each test separately. Data analysis requires development of custom reports within web analytics tool.
In practice, use of web analytics for enhanced test result analysis is very cumbersome. It’s use is more of an exception rather than the best practice.
At the beginning of the test browser analysis can reveal technical problems with the test treatments. At the end of the test it is possible to uncover treatment preferences by visitors using different browsers:
Visitors who use Chrome prefer Variation 2 – Featured Pro: +9.90%
Target Chrome users with Variation 2 – Featured Pro treatment
This is not the impression you will get by reading Which Test Won.
They are not telling you that in the pool of tens of thousands of prospectors (testers) running hundreds of thousands of test somebody will get lucky and then brag about it. That’s perfectly fine, for as long as we all know that this is an exception and not the norm.
We hope that this article will inspire to take a more systematic approach to testing.
By arming your testing solution with rich web analytics you will be able to extract a few golden pieces from each test and apply learning to continually improve the quality of your test treatments.