A conventional wisdom is to run A/B or Multivariate tests during pre-season. Then, during the holidays do nothing. Just sit and enjoy the benefits of your prior hard work. The few brave souls that extended tests into the holiday season will tell you that the conventional wisdom is wrong.

High Impact of High Season

There is no formal definition of the holiday season. Generally speaking the period from early in November until the end of December when the volume of online traffic and sales are up is considered the holiday season.

The season is a big deal and it gets a lot of attention. Approximately at this time in the New Year we all go through the ritual of reporting on double digit growth of online sales:


Some analysts would go a step further and segment the holiday season into sub intervals. For example, Slice Intelligence has created the following report:


Holiday sales are a significant percentage of annual sales for almost every type of online business:


During the holiday season stakes in the game are high. A wrong move can cost you a lot of money.

Should You Test During The Holiday Season?

Although the majority of online brands will respond with a resounding ‘no’ let us for the sake of completeness list pro and con arguments.

PRO: Testing During Holiday Season Is Not Optional

If we would follow the logic of the famous bank robber William Sutton who once said that he is robbing banks “Because that’s where the money is,” we would say that one should test during the holiday season because of the huge potential upside.

There is a lot of stuff that can go right …

  1. An increase in the number of visitors increases the speed of testing.
  2. Average order value is higher making revenue lift easier to achieve.
  3. Conversions are up making it easier to get statistically significant results.
  4. Variety of visitors is up making it possible to generate real new revenues.

Testing is the essential tool for engaging and learning about your customers. Test treatments, even those that do not provide a lift, teach us about customer preferences which on the long run will produce sustainable revenue growth.

CON: It’s Too Risky

Following the same bank robbers analogy one can argue that testing during high season is too risky. You can get caught and end up in ‘jail.’

There is a lot of stuff that can go wrong …

  1. Spikes in traffic are skewing results while creating fluctuation of numbers from one day to another.
  2. Promotions are reducing average order values and introducing negative pressure on revenue growth.
  3. Testing is risky. Your good intentions can lead to a drop in conversion rate. This will make your boss very unhappy.

A Conventional Wisdom: Test During Pre-Season

Since testing during the holiday season is too risky eCommerce brands typically stop testing during the holiday season.

Code freeze and moratorium on any site changes are the norm.

Instead, all testing and site improvements are done during the pre-season.

The focus is on a general customer experience and on getting the site ready for seasonal shoppers:

Web Analytics and Heat Maps: An in-depth data analysis will uncover friction points in the conversion funnel.  Use of click mapping tools like HiConversion will help with data driven test designs.

Usability Studies And Reviews: Usability studies will create an inventory of great test ides based on comparison of your site with other third-party sites. What are people saying about your site? Use of review tools provides another layer of super valuable input.

Customer Experience Optimization: Test ideas from above are then used to experiment with different areas of the site, such as checkout, product detail pages, navigation, etc. Perfect layout, style, copy and above all a seasonal value proposition. The goal is to differentiate your offering and create an enjoyable online shopping experience.

One Problem: The Conventional Wisdom Doesn’t Work 

First, how much we really know about visitor behavior during the holiday season? And, how useful are insights we get from traditional web analytics solutions?

Fatal Assumption

Driven by fear, brands are making a big assumption that visitors behave in the same way during the holiday season as they do in preseason. Therefore, it is very logical to optimize during the preseason and reap the big benefits during the high season.

As you will see from the examples below visitor behavior during holiday season is different than in the pre-season. The sources of different behavior include visitor motivation, incentives, need for speed and simplicity, etc.

No Insights

By not testing online brands are actually turning the lights off.

Since prior test results are fully implemented nobody knows how preseason site improvements are actually working during holidays.

Examination of web analytics reports tells you nothing. Year-over-year revenue growth is obfuscating the sources of that growth.

Can we attribute such results to site improvements or will revenue grow even more if site improvements were not made.

If you are not testing during the season nobody will ever know.

We were fortunate to have many of our clients listen to our advice and test all the time. What follows are a couple of examples of different behavior during and after the season.

Example #1: Preseason Test Results Are Negative

The following is an example of the Product Detail Page (PDP) Multivariate optimization test done by a well known brand at one of many regional eCommerce sites.

The test started during the pre-season and then it was continued throughout holiday and post-holiday periods.

Test Variables

Multivariate testing can be viewed as a group of A/B tests running at the same time.

To perform such a test one needs to use a testing tool that supports such capability while providing insight into individual results of each A/B test as well as an ability to show how combinations of treatments performed as a group.

This test included the following treatments:

  • Cross Sells: move product recommendations higher on the page
  • Photo Share:  make user created content more prominent
  • Mini Cart: modify layout and the content of the mini cart
  • Distractions: remove several auxiliary elements like reviews or social links from the product selection area

Test Results

The picture below shows the cumulative test results for the entire duration of the test. Sample size, confidence, and lift look good.


Furthermore, considering that the test ran over an extended period of time one would also conclude that these results are ‘evergreen’ results that should work all the time.

A common practice at this stage will be to recommend a permanent implementation of the winning treatments.

Time As A Segment

HiConversion’s optimization technology is built on the premise that visitor behavior is always changing and that the only sustainable solution is in the ability to automatically detect and adapt to changes in real time.

That is why in the analysis of the test results we always go beyond cumulative results and statistical significance.

In this example we segmented time intervals into 3 sections:


Please notice that during the test itself this client obtained an overall lift of +8.16. Meaning that they made money while running the test.

Time Segment#1: Preseason

To our surprise the test results in the beginning did not look promising.


When a client voiced concerns and a desire to stop the test we advised a little bit more patience.

And we are glad we did it.

Time Segment#2: High Holiday Season

The patience started to pay off.

During the busiest period of the holiday season the same losing treatments performed extremely well:


Time Segment#3: Postseason

Unfortunately, as soon as holiday season was over the numbers went south again.


So,What To Do With This Test?

The analysis above makes it very clear that the holiday season has skewed the test result. Something that looked as a solid winner before this analysis now looks as a total loser.

In advising clients  it is quite convenient to ignore this analysis and make recommendations based on the overall results. After all, once ‘winning’ treatments are implemented nobody can detect that postseason results are actually negative and that they will negatively impact future revenue lift.

A viable (and ethical) solution is to educate clients about the time varying nature of results. Our experience is that this impresses our clients more than any conversion lift.

Secondly, the test is never a ‘loser’ if you leverage insights provided by  customer experience analytics to identify persuadable segments that consistently performed across extended time intervals.

Example #2: Preseason Test Results Are Positive

In the first example we had a case where preseason test results did not look good. We were lucky to convince our client to stay put and persist with the test.

Let us now share an example of the opposite case.

Test Variables

This was a simple A/B/n test of the main navigation drop down menu.

This test included the following treatments:

  • Variation 1 – GO: less expensive GO product image featured in the mega drop down
  • Variation 2 – PRO:  image of the high end PRO version of the product in the mega drop down

Test Results

Similar to the first example above let us examine the results during three time segments.

Time Segment#1: Preseason

After 8 weeks preceding November 24 the client had two positive test results. GO version was significantly better then PRO.


Time Segment#2: Holiday Season

If client implemented the GO product they would miss out on learning that the holiday crowd preferred the PRO product and to sell it at much higher rate than GO.


Time Segment#3: Postseason

During the weeks that followed the holiday season the performance of GO product has further eroded.


At this stage the presence of the preseason winning GO treatment would actually start to drag down the revenue performance of the site.

If our client did not run this test over different seasonal time intervals nobody would ever detect this.


The time varying nature of test results is both a challenge and an opportunity.

The challenge can be solved through the use of adaptive algorithms that have the ability to detect and adapt to visitor preferences in real time.

By embracing instead of ignoring this time varying nature of visitors brands will gain a unique competitive advantage.