Nate Silver leverages the power of data to accurately predict US presidential elections on a state-by-state basis. Now, he’s back to predict the Super Bowl winner. In this post, we’ll share his methods with you, and show you how you can use these same methods to make better decisions about your e-Commerce site.
- Data matters
- There are many factors at play
- It is about probabilities – nobody can be 100% right all the time
In the recent presidential election, Nate Silver was able to build a statistical model to correctly predict who will win each of the 50 states – and ultimately – the presidential election. Now he’s back, this time using historical data about the outcomes of past Super Bowls to illustrate that defense wins championships.
Though the Ravens pulled out the upset, what really caught my attention was not who should win, but rather the thought process that goes into his analysis, since it’s in-sync with our eCommerce optimization methodology and provides a framework that can benefit e-Commerce professionals, regardless of whether their pick won last night.
Nate began his short analysis by posing a question: does defense win championships?
To answer that question, Nate starts by digging into the available statistics. In his article, Nate mentions Simple Rating System (or S.R.S.) statistics, which evaluates each team’s offense and defense based on the number of points it scored and allowed relative to the league average, and then adjusts for strength of schedule.
The list below shows the best defensive teams that played in the Super Bowl:
Image Credit: NYTimes
Based on that data alone, the probability of the stronger defensive team winning is 70%. However, Nate goes one step further to exclude historical games between two ‘equally-strong’ defensive teams, and states that a stronger defense is a predictor of winning 80% of the time.
But don’t discount offensive strength
Nate proceeds to evaluate stats for the teams with strong offense. Here’s the list:
Image Credit: NYTimes
A similar analysis reveals that the top offense has only a 50% chance to win. In this way, the 49ers were the winning favorite: they had both the better defense and offense.
What about other factors?
Nate suggests that a more sophisticated analysis should also take into account data published by Football Outsiders. Their system, known as Defense-adjusted Value Over Average (D.V.O.A.), accounts for not only final scores, but the result of each play that a given team ran. In other words, there is a need to account for variables, including the impact of the special teams, and interaction(s) between different aspects of the game. Only then, he argues, will you have a statistical model capable of predicting future outcomes with good reliability.
The e-Commerce connection
Improving your e-Commerce site’s performance is a process that involves making calculated bets on actions that should produce a positive ROI. In order to succeed, you should first bet on items that have a high probability of success.
Our simple analysis of Nate’s thought process can help you to create your own framework for improving your e-Commerce site:
Know Your Data: In Nate’s line of business, the datasets already exist. All he had to do was research and then apply the data to his number-crunching algorithms. In your case, you might not have all the data you need about your e-Commerce site performance. For example, before making a decision about whether to invest in a recommendation engine, you’re unlikely to have all the data you need to estimate its potential impact on your visitors’ behavior. Your suppliers will share stats from other clients, but that may or may not be useful.
The best way is to collect your own data. Develop systems that will measure the impact of whatever you do on your eCommerce site. Only then you will be empowered to make a data driven decision.
Evaluate Multiple Aspects: Note that Nate did not use insight about the strength of a team’s defense to immediately jump into conclusions about which team was going to win.
Often times, companies are running single-variable A/B tests and jumping to conclusions prematurely based on their results. What Nate suggests is that you have to include other variables; in e-Commerce, these variables can be content variables, layout changes, web applications and widgets, and more.
Nothing is 100% Predictable: In Nate’s article, he illustrates that (statistically-speaking), the 49ers had an advantage in both aspects of the game, defense and offense, and as such were the winning favorite. However, he’s careful not to state that they will actually win (we now know they didn’t!)
His approach, similar to our adaptive optimization methodology, works because it considers multiple angles to give you the highest probability for success and lowest chance for incurring risk. Any short-term surprises are likely to average themselves out, leaving you with a nice profit from whatever you do with your e-Commerce site.
Finally, the outcome
Upsets happen: despite having an advantage “on paper”, the Ravens won the Super Bowl. It turns out that special teams – with a 108-yard kickoff return for a touchdown – was the variable that made the difference.
One key aspect of statistical analysis that Nate neglected to mention was the time-varying nature of team performance indicators. For example, the 49ers defense was not as effective as they were during the regular season, and there are a potentially limitless number of contributing factors to explain why.
When it comes to e-Commerce, the time-varying nature of purchasing behavior can quickly render any static analysis obsolete. In order to win in e-Commerce, you need an adaptive technology that can detect and leverage changes in your visitors’ behavior in real time.