From analytics to testing and even personalization, e-Commerce optimization is built on the practice of collecting and interpreting statistically significant results. However, there’s a big catch: yesterday’s results don’t quite guarantee future outcomes. So what do you need to know in order to ensure that your e-Commerce metrics are telling the whole story?

Key Takeaways:

  • Even statistically significant results can be misleading
  • Visitor behavior is always changing
  • You can use time-varying shopping behavior to your advantage

Numbers don’t lie, or do they?

Simply put, e-Commerce is a numbers game: in order to succeed, you must correctly align your revenue to expense ratios. To accomplish this, you’ve got to accurately measure everything you do, and then make data-driven decisions.

Analytics, testing, and personalization vendors will all testify that the road to success starts with achieving statistically significant results. On the surface, this makes sense. For example, the table below shows results from a multivariate test. These results indicate that one version of the page (Combination 42) had a +65.76% lift in revenue per visit (RPV) against the baseline.

e-Optimizer® statistical results for various combinationsrevenue growth

Further, the page was displayed 9,057 times and converted visitors on 3,115 occasions. Based on those numbers, most e-Commerce professionals would implement the winning combination and expect – within reason – that the outcome wouldn’t deviate much from those original results.

Statistical confidence is important, but it’s not the determining factor

Over the years, we’ve spoken to hundreds of e-Commerce professionals who hail from dozens of industries. Within this group are a wide variety of personalities from many different types of organizations. Yet, they all have something in common: most of them have (at one time or another) run back-to-back testing or targeting campaigns on their e-Commerce sites. Each campaign they run uncovers something interesting or worth implementing; yet time after time, the actual outcomes don’t seem to add up.

The test results looked great, but where’s the revenue growth?

One major disconnect is an over-reliance on statistical confidence. In reality, statistical confidence does not tell the full story. Behind that great +65.76% RPV lift from the previous example lies a picture that demonstrates real time purchasing patterns.

e-Optimizer® line graph comparing combination 42 vs baseline

Looking at the line graph above, it quickly becomes apparent that Combination 42 was not significantly better (or worse) than the baseline most of the time. However, there was a short period of time – approximately 10 days – wherein the performance improved dramatically. This could have been the result of a promotional campaign by either the client or their competitors, or the time of year, e-mail drops, and a variety of other outside factors.

Both of these sets of data represent the same results. The difference is in how you view them.

True – statistical significance is important, but it certainly does not tell the whole story. Notice how Combination 42 began to underperform towards the end of the test: based on the results from the table, would your outcomes have been as expected, had you implemented this combination permanently?

The implications

Many e-Commerce pros don’t respect the magnitude of change in visitor behavior over time: they think that any fluctuations in purchasing patterns are small relative to some moving average. In fact, the magnitude of change is significantly large – we observe changes in client performance to the tune of 5-10x over the moving average on a daily basis.

Time-varying behavior creates both a problem and an opportunity for e-Commerce. If you rely simply on test results or rules-based personalization solutions to drive your decisions, you’ve applied a static solution to a dynamic behavior problem, and the success rate can be understandably low.

There is a way, however, to leverage time-varying behavior patterns and use them to grow your revenue. With real-time optimization technology, you can adapt to visitor behavior and adjust traffic proportionately to how your pages are performing, thus maximizing your potential to increase RPV and grow online revenue.

Footnote: The title of this blog post, “Lies, damned lies, and [e-Commerce] statistics” comes from the phrase that describes the persuasive power of numbers, particularly in the use of statistics to boost weak arguments. The term was popularlized by Mark Twain who attributed it to the 19th-century British Prime Minister Benjamin Disraeli. “There are three kinds of lies: lies, damned lies, and statistics.”