You can segment your traffic, target your audience, personalize your message, and show your visitors what you think they may prefer to see – but ultimately, these are all a gamble. In this post, we’ll show you how to remove this risk by incorporating some intelligent strategies that will help you know when to bet (and when to fold) before your cards are dealt.

Key Takeaways:

  • Traditional testing and personalization is a big gamble.
  • Predictive optimization eliminates risk – and it isn’t even cheating.
  • You can have your cake and eat it, too.

Vegas, baby, Vegas!

Las Vegas is the only place I know where money talks – it says, ‘Goodbye.’

– Frank Sinatra, The Joker is Wild (1957)

Well said, Frank.

Tomorrow, my colleagues and I are headed to Las Vegas to attend the 2013 Demandware XChange conference – an event that we’ve sponsored for the past three years. Of course, the great location has me thinking – not simply about where I’ll lay down some chips, but also what better context for a gathering of the world’s top online retailers than the Vegas strip?

Yup, you heard me right: what better place to gather hundreds of revenue-driven e-Commerce executives than the city where millions flock with dreams of hitting the jackpot?

After all, many of us gamble every day. For some, it’s baked right into your job description. This type of gambling doesn’t involve red or black, horses, cards, or chips; rather, it’s the game of chance we label online testing.

By launching a new A/B or multivariate test, you’re betting that a certain version of your website or a certain combination of changes to that website will perform better than what’s already there. Often, ideas for these tests come from our gut, industry best practices, or previous experiments. Isolated in a vaccum of time, this idea makes sense – but there’s a problem – your test may not work, and more importantly, is all-but-guaranteed not to work sustainably over time.

The problem: your shoppers’ preferences change (a lot)

Time-varying behavior is nothing new: we’ve written extensively on the subject. The disconnect is the magnitude of variance. In other words, most agree that visitor behavior changes over time, but most e-Commerce pros have no idea just how much variance there is.

At HiConversion, we have benefit of years of data from lots of clients across dozens of industries, and in that data are signs that variance is not only commonplace, but the order of magnitude can be 1-10x per day. This means that each time you launch a new test, you’re betting real dollars that the number of times the new version performs better than your control is greater than the number of times it performs worse. The illustration below demonstrates how each time your test performs worse, you’re losing revenue-generating potential.

potential revenue loss over time from testing

With traditional testing, the cards are stacked against you, and you’d be lucky to break even. Many actually lose money during the test phase, but assume that they’ll make it up in the future once the winner is implemented. True, you’re behind – but you’ve got a good hand that you’ve just yet to play, right?

So you show your cards, but the house wins again.

What happens during the test phase stays in the test phase

The challenge with implementing winners is that they’re based on statistically-significant results from a past time interval. The notion that statistically-significant results are valid predictors of future outcomes is fatally flawed because it’s based on the assumption that your visitors’ behavior doesn’t change; in reality, that’s almost never the case.

In other words, by the time you’ve implemented your winners, they’re probably obsolete.

Your visitors have moved on. You should, too.

How to get the upper hand

First, know that ‘card-counting’ is legal in e-Commerce

e-Commerce managers are notorious for their bad poker face. As soon as they reach significance, most jump to implement the winner. Some even have solutions that will automatically perform this. Other, more skeptical guys and gals wait a certain amount of time (prospects usually reveal an average limbo of about 7-14 days after significance has been reached) to implement. In the latter case, they recognize the potential for variance and want to wait to be sure; what they’re actually doing, however, is extending the test phase and therefore increasing their liability for profit loss. So skepticism is a good thing – it indicates that you don’t trust the winners’ luck to hold – but unfortunately, there’s very little you can do within the framework of traditional A/B and MVT to mitigate the risks of online testing.

Remember the MIT Blackjack Team that used card-counting techniques to win money? Statistical modeling and prediction can be a very powerful tool – even something as random as a game of blackjack exhibits some predictable trends. So why not use such a capability in e-Commerce?

Know when to fold (and when to bet) before your cards are dealt

You can’t sustainably or predictably win with traditional online testing, but you can change the way you think about testing and optimization by becoming agile.

Advanced optimization solutions (plug alert: like e-Optimizer®) give you the ability to adapt to your visitors’ behavior in real time. It allows you to be able to predict whether a creative (e.g., a banner, an offer, a layout, or functionality features) will beat the house, or whether to fold and show your baseline or some other version before your visitors arrive and your cards are dealt.

If you used this strategy in Vegas, you might end up in a casino basement. But in the world of e-Commerce, predictive modeling is your ticket to sustainable revenue growth.

And while I don’t particularly like to gamble, I do love to win.

More on predictive modeling and adaptive algorithms

Like the Vegas strip itself, eCommerce technology and metrics have come a long way in a very short period of time. At the bleeding edge are solutions that harness the power of artificial intelligence and modeling to quickly and accurately predict the outcomes of multivariate tests.

These algorithms know when to play defensive and when to display your test copy, images, offers, and other variables. By playing more of what works – and consequently, less of what doesn’t  – across multiple pages on your e-Commerce site and for multiple segments simultaneously, you can effectively leverage your visitors’ time-varying behavior patterns to create a positive lift to revenue.


As opposed to online testing and personalization, adaptive optimization isn’t something you’d like to turn off (after all, it’s making you money). In fact, this is where we derive the difference between testing and optimization. The former is tactical and risk-intensive, and the latter is strategic and lends itself to revenue-positive results.

I’m looking forward to our upcoming trip to Demandware’s XChange conference, where we’ll meet with both present-day and future clients to talk e-Commerce optimization.

Sadly, though, we do have one disclaimer: don’t try to use e-Optimizer® at the blackjack table (for that, you’ll need to submit a feature request).