A conventional wisdom is to run A/B or Multivariate tests during pre-season. Then, during the holidays do nothing. Just sit and enjoy the benefits of your prior hard work. The few brave souls that extended tests into the holiday season will tell you that the conventional wisdom is wrong.
There is no formal definition of the holiday season. Generally speaking the period from early in November until the end of December when the volume of online traffic and sales are up is considered the holiday season.
The season is a big deal and it gets a lot of attention. Approximately at this time in the New Year we all go through the ritual of reporting on double digit growth of online sales:
Some analysts would go a step further and segment the holiday season into sub intervals. For example, Slice Intelligence has created the following report:
Holiday sales are a significant percentage of annual sales for almost every type of online business:
During the holiday season stakes in the game are high. A wrong move can cost you a lot of money.
Although the majority of online brands will respond with a resounding ‘no’ let us for the sake of completeness list pro and con arguments.
PRO: Testing During Holiday Season Is Not Optional
If we would follow the logic of the famous bank robber William Sutton who once said that he is robbing banks “Because that’s where the money is,” we would say that one should test during the holiday season because of the huge potential upside.
There is a lot of stuff that can go right …
An increase in the number of visitors increases the speed of testing.
Average order value is higher making revenue lift easier to achieve.
Conversions are up making it easier to get statistically significant results.
Variety of visitors is up making it possible to generate real new revenues.
Testing is the essential tool for engaging and learning about your customers. Test treatments, even those that do not provide a lift, teach us about customer preferences which on the long run will produce sustainable revenue growth.
CON: It’s Too Risky
Following the same bank robbers analogy one can argue that testing during high season is too risky. You can get caught and end up in ‘jail.’
There is a lot of stuff that can go wrong …
Spikes in traffic are skewing results while creating fluctuation of numbers from one day to another.
Promotions are reducing average order values and introducing negative pressure on revenue growth.
Testing is risky. Your good intentions can lead to a drop in conversion rate. This will make your boss very unhappy.
Since testing during the holiday season is too risky eCommerce brands typically stop testing during the holiday season.
Code freeze and moratorium on any site changes are the norm.
Instead, all testing and site improvements are done during the pre-season.
The focus is on a general customer experience and on getting the site ready for seasonal shoppers:
Web Analytics and Heat Maps: An in-depth data analysis will uncover friction points in the conversion funnel. Use of click mapping tools like HiConversion will help with data driven test designs.
Usability Studies And Reviews: Usability studies will create an inventory of great test ides based on comparison of your site with other third-party sites. What are people saying about your site? Use of review tools provides another layer of super valuable input.
Customer Experience Optimization: Test ideas from above are then used to experiment with different areas of the site, such as checkout, product detail pages, navigation, etc. Perfect layout, style, copy and above all a seasonal value proposition. The goal is to differentiate your offering and create an enjoyable online shopping experience.
First, how much we really know about visitor behavior during the holiday season? And, how useful are insights we get from traditional web analytics solutions?
Driven by fear, brands are making a big assumption that visitors behave in the same way during the holiday season as they do in preseason. Therefore, it is very logical to optimize during the preseason and reap the big benefits during the high season.
As you will see from the examples below visitor behavior during holiday season is different than in the pre-season. The sources of different behavior include visitor motivation, incentives, need for speed and simplicity, etc.
By not testing online brands are actually turning the lights off.
Since prior test results are fully implemented nobody knows how preseason site improvements are actually working during holidays.
Examination of web analytics reports tells you nothing. Year-over-year revenue growth is obfuscating the sources of that growth.
Can we attribute such results to site improvements or will revenue grow even more if site improvements were not made.
If you are not testing during the season nobody will ever know.
We were fortunate to have many of our clients listen to our advice and test all the time. What follows are a couple of examples of different behavior during and after the season.
The following is an example of the Product Detail Page (PDP) Multivariate optimization test done by a well known brand at one of many regional eCommerce sites.
The test started during the pre-season and then it was continued throughout holiday and post-holiday periods.
Multivariate testing can be viewed as a group of A/B tests running at the same time.
To perform such a test one needs to use a testing tool that supports such capability while providing insight into individual results of each A/B test as well as an ability to show how combinations of treatments performed as a group.
This test included the following treatments:
Cross Sells: move product recommendations higher on the page
Photo Share: make user created content more prominent
Mini Cart: modify layout and the content of the mini cart
Distractions: remove several auxiliary elements like reviews or social links from the product selection area
The picture below shows the cumulative test results for the entire duration of the test. Sample size, confidence, and lift look good.
Furthermore, considering that the test ran over an extended period of time one would also conclude that these results are ‘evergreen’ results that should work all the time.
A common practice at this stage will be to recommend a permanent implementation of the winning treatments.
Time As A Segment
HiConversion’s optimization technology is built on the premise that visitor behavior is always changing and that the only sustainable solution is in the ability to automatically detect and adapt to changes in real time.
That is why in the analysis of the test results we always go beyond cumulative results and statistical significance.
In this example we segmented time intervals into 3 sections:
Please notice that during the test itself this client obtained an overall lift of +8.16. Meaning that they made money while running the test.
Time Segment#1: Preseason
To our surprise the test results in the beginning did not look promising.
When a client voiced concerns and a desire to stop the test we advised a little bit more patience.
And we are glad we did it.
Time Segment#2: High Holiday Season
The patience started to pay off.
During the busiest period of the holiday season the same losing treatments performed extremely well:
Time Segment#3: Postseason
Unfortunately, as soon as holiday season was over the numbers went south again.
So,What To Do With This Test?
The analysis above makes it very clear that the holiday season has skewed the test result. Something that looked as a solid winner before this analysis now looks as a total loser.
In advising clients it is quite convenient to ignore this analysis and make recommendations based on the overall results. After all, once ‘winning’ treatments are implemented nobody can detect that postseason results are actually negative and that they will negatively impact future revenue lift.
A viable (and ethical) solution is to educate clients about the time varying nature of results. Our experience is that this impresses our clients more than any conversion lift.