In e-Commerce business, one has two options to grow revenue. The first one is to get more visitors to the site, which is increasingly harder and more expensive. The second one is to optimize visitor experience on the site itself and generate more revenue from the existing traffic.

The process of getting more from the existing traffic is broadly called Customer Experience Optimization (CXO).

There are many types of software solutions that assist with optimum customer experience including – web analytics, A-B and MVT testing, targeting, re-targeting, personalization, etc. Vendor claims and market best practices are all over the map and this has affected effective adoption of CXO solutions.

The purpose of this post and a few others that will follow, is to go back to basics and to help online professionals understand the nature of the problem so that they can become more educated consumers of the available solutions.

It is not about big data, it is about the BIG number of options

e-Commerce is super exciting because of its interactive nature and the ability to track visitors or take actions that produce measurable results.

When it comes to the data side, there is a lot of buzz about the use of big data technologies to effectively collect and process unimaginable numbers of visitor online data.

It is counter intuitive, but the value and the competitive advantage does not come from the ability to collect and sort more visitor data. The most significant value will come from the ability of the optimization tool to deal with the huge number of options made possible by bigger data sets.

Data world is not flat

Visitor and experience attributes can be grouped into more understandable data sets:



Traditional web analytics tools are focused on analysis of visitor attributes with the goal of creating meaningful visitor/customer types.

The number of visitor attributes is huge:


– Marketing channels (SEO, PPC, e-mail, social, …)

– Devices (desktop, mobile, tablet, …)

– Geo locations (countries, regions, metro, city, …)

– On-site behavior …

For example, the other day we were told about a 6-month project done by a boutique web analytics firm where the mission was to process terabytes of visitor data and parse through approximately 1,600 visitor attributes, to identify the dominant visitor types in the pool of trillions of multi-attribute options . They were super happy to report that such efforts uncovered approximately a dozen dominant visitor types. Very impressive stuff!


Knowing dominant visitor types is just a step in the right direction. The next challenge is to get a grip on the visitor consideration path. We call it the buying journey.

e-CXO JourneyThat buying journey can start at different entry points and can take many twist and turns along the way. People who have tried to do path analysis, understand that visitors can travel through an e-Commerce site in millions of ways.

Similar to the need to discover dominant multi-attribute visitor types, one needs to uncover dominant multi-step buying paths. Each of the buying journeys may require a different web page experience.

On-Page Experiences

OnPgExpThe physical on-page experience will directly influence reactions of different visitors at different steps of the buying journey.

Each page has dozens of key elements and each element can be presented in a wide variety ways. The on-page changes are driven by desire to support value proposition, to enhance or better define call-to-action, to create more pleasant visual appearance, or to create urgency or ongoing engagement.
Permutations of the element variations will result in millions of page versions and the challenge is to find which version of the page performs the best.

Checkerboard Example

Mentioning big numbers of multi-attribute visitor types, multi-step buying journeys, or millions of web page variations, still falls short of making it more obvious that customer experience optimization is a gigantic problem.

To even better illustrate this point, let us compare the Customer Experience Optimization to a checkerboard.

Let’s start with the objective to optimize an e-Commerce site where buying journey has 8 steps (pages) in the sales funnel and let’s imagine that we desire to treat 8 elements of each page with only one variation. This sounds like a reasonable objective.

We can visually represent this problem with a checkerboard that has 8×8 size, representing eight steps and eight variable elements at each step, with black field color for the variation and white field color for the control (the existing element on the physical web page).

If we assume that the alternating black and white pattern is the optimum solution that provides the highest revenue lift then the logical question is – how many permutations of the checkerboard pattern are there? The answer is 264

2^64: The number of combinations from a checkerboard

Even this problem representation is hard to translate in to something we can all feel with our own human senses. So, let us transform this big number in number of kernels of wheat:


Eighteen quintillion, four hundred and forty-six quadrillion, seven hundred and forty-four trillion, seventy-three billion, seven hundred and nine million, six hundred thousand!

This number is so big that humankind has not produced this much wheat since the beginning of time!

In other words, if we translate the problem of finding the optimum customer experience in the pool of 8 web pages with 8 elements with 1 change, we are talking about finding a kernel of wheat in a pile of wheat that is bigger than the entire global production of wheat in human history.

If all you have is a hammer, everything looks like a nail …

Now that we know that the optimization of customer experience is such an extraordinarily big challenge, the logical question is what ‘medications’ can be used to ‘cure’ this issue.

The marketplace is dominated by A-B testing solutions. Even premium testing and personalization solutions that differentiate their offerings through MVT capabilities are not making any difference. Based on our daily experience with companies that paid a premium for MVT capabilities, it is increasingly obvious that they are mostly running only simple A-B test.

When asked why they would not run MVT all the time, their answer was that they actually did 1-2 MVT campaigns only to learn that the MVT solution was too complicated to setup and too slow to produce results.

This thinking is consistent with vendor recommendations (see Optimizely’s blog below):

The single biggest limitation of multivariate testing is the amount of traffic needed to complete the test. Since all experiments are fully factorial, too many changing elements at once can quickly add up to a very large number of possible combinations that must be tested. Even a site with fairly high traffic might have trouble completing a test with more than 25 combinations in a feasible amount of time.

In closing

The sheer size of the optimization problem is just a starting point that indicates that traditional testing solutions are ill-equipped to deal with the monumental task of improving results through customer experience optimization.

And the complexity of the e-Commerce optimization problem does not stop at the size of the problem. In posts to follow, we will reveal the importance of interconnections between elements of the customers experience as well as time varying nature of visitor preferences.