Testing and personalization is a new ‘shiny’ digital marketing solution for digital commerce brands and for all good reasons. It is common sense that more personalized customer experiences convert better.

Testing and personalization vendors are talking the talk, but can they walk the walk?

The old ‘black color’

At it’s most basic level, digital commerce is about driving traffic to the site and creating good customer experiences that convert and make money.



The challenges

Demand generation is increasingly becoming more and more commoditized. Digital brands are competing among themselves for the same traffic. This is driving cost up and return on investments down. The ability to track and measure the quality of such traffic is limited and everybody is at the mercy of uncontrollable external factors: like competitor’s promotions, social interactions, news, stock market, etc.


Internally, digital commerce is impacted by the work of many teams: from UX people who are making changes to the look and feel of your site, commerce teams that are adding new features and applications, to marketing teams that are running promotions, etc. These non-coordinated activities are adding more fragmentation and friction to already fragmented and non-optimum customer experiences.

The combination of the external and internal factors is putting digital brands in a volatile and unpredictable revenue situation.

The industry stats, like average conversion rate that is under 2% or 70% cart abandonment rate are showing the true extent of the customer experience problem.

Can testing and personalization help?

Yes. But only if your testing and personalization solution can handle all this:

Visitor segmentation: Web traffic is like a river. Water flows through the same place but at each moment the level of the water, the kind of fish, the number of fish, speed, temperature, clarity, etc. is different. Making the assumption that ‘if-this-audience-then-do-this’ personalization will consistently work is like always fishing at the same location and expecting to catch the same fish like you did once before.

Visitor preferences: Staying with the river and fishing analogy, a testing and personalization solution must be able to detect not only if there is fish (type of visitor) but also specific preferences of the fish in the water and provide the ‘bait’ that converts. Expert fly fishermen will tell you that fish preferences are constantly changing and that that one needs to constantly experiment and adapt if they expect to catch.

Whole experience: The act of catching fish, i.e. making online sales, dervies from many elements that depend on one another. The current approach where one tool is independently recommending products while the other tool is targeting visitors with a promotion is the equivalent of two fishermen, one of whom has the right bait, while the other one has the right spot. Neither one will catch fish until they unify efforts and get the right bait presented at the right spot.

The new black

If testing and personalization solutions can simultaneously handle different visitors, time varying preferences, and huge number of variables then such a solution should be called by a different name: adaptive customer experience optimization.


The adaptive part is the key new capability.

The existing testing and personalization offerings are fragmented for a reason: they are not capable of handling huge numbers of everchanging variables in real time.

Marketing departments of testing and personalization vendors are very successful at getting attention drawn to the potential benefits of their solution. They are likewise excellent at hiding the dark side:

  • They never talk about the inherent risk of testing. Namely, while experimenting with different options, the probability is extremely high that the number of losers will be higher than the number of winners, which implies, if not guarantees that experimentation without intelligence will result in a dip in conversion and loss of revenue for the duration of the test period.
  • They are not talking about the time varying nature of digital commerce and that test results have a limited shelf life. So, that even after you went through the pain of losing money while experimenting to find a winning test result, that you still need to be very concerned about how long the winner will hold up… if at all.
  • And they are certainly not telling you that every element of the customer experience is linked with every other element of the experience. If you break one you break the whole experience. They want you to believe that running hundreds of A-B tests or rules based ‘if-this-do-that’ targeting campaigns is good for your brand. Where in fact, if you do not align the right bait with the right spot you will never catch a fish. Such alignment is only possible if the solution can handle large numbers of variables at ulta high speed.

Should you wear it?

Yes and in many ways.

If you are already committed to a testing and personalization platform then you should consider practices that minimize the damage:

  • Replace A-B testing with multivariate testing
  • Post-validate test results to make sure that they are stable with long term viability
  • Establish an environment that enables proper site wide measurement of the impact of the testing and personalization variables – one key enabler of accurate results tracking is the ability to establish a site wide control group of visitors who are presented by the non-changed site so that you can know what would happen if you did not do anything

If you are currently not using any testing and personalization solution you should at bare minimum consider use of customer experience web analytics. Knowledge about customer experience friction points will help your brand to allocate resources and commit to the adaptive customer experience optimization.

Create FREE Account
* No Credit Card Required