Correlation vs Causation and Its Impact on Site Testing

"Correlation does not imply causation."

Heard that phrase lately? Or maybe way back in Stats 101?

The topic is so hot it got it's own panel at Search Marketing Expo and a math-o-graphic on the SEOMoz blog. It's worth discussing here on Get Elastic as it's a key truth that applies to data analysis and A/B or multivariate testing.

I'll give you a minute to look this over...

Got it?

https://xkcd.com/552/

Let's apply this to ecommerce.

Last June, Canada Post went on a lengthy strike/lockout, and many ecommerce sites around the world were affected, as were consumers. On my iddy-biddy Etsy site, I saw a significant drop in orders - 30% down from the previous month, and 40% less than the following month. Pop quiz - is this correlation or causation?

Sounds like causation, right? Well, I think it's safe to say that a carrier strike most definitely prevents sales which negatively impacts revenue. No doubt about it. But is it correct to quantify the strike as having a 30-40% impact on sales?

Nope. There are many other variables that may have impacted my May, June and July sales - June is post-Mother's Day and gifting attention turns to Dads n' Grads, for which my product line is less of a fit. Etsy "front page" and email newsletter features that may have occurred during these months have a major impact on traffic and sales and are serendipitous, outside my control. There may have been a change in Etsy's search algorithm that affected my shop items. I change my merchandising and added a new line of cell phone cases in June - an increase in items for sale should correspond with an increase of sales.

The point is, without considering these variables, I may wrongly conclude that the interruption of postal service was responsible for such a large percentage of sales. Replace "postal service" with "website redesign" or "10% off promotion" or "site outage" and you get the idea - we operate our ecommerce marketing and optimization programs in a vast ecosystem of variables and uncontrollable and unmeasurable events. It's very, very, very difficult to create an experiment that controls for all the factors that may be influencing our performance metrics.

So, what's a girl to do?

• Continue site testing, tweaking, measuring and making decisions - but embrace these as correlative and avoid drawing causative conclusions and these fallacious conclusions to other marketing activities, campaigns, websites or businesses you may be working with.

• When A/B or multivariate testing, be careful to reduce bias as much as possible. This means planning tests around web launches, major IT changes/upgrades, certain holidays and so on. It also means being careful not to run tests on different sections of a site concurrently that can introduce experiential bias, such as the combination of showing a free shipping banner on the home page and testing a one-page checkout. For example, a customer motivated by free shipping may work harder with a clunky system than one who is being charged for shipping and has many alternative sites to shop with.

• Consider repeating certain experiments to validate whether the "evidence" holds up over time or under different market conditions, seasonal cycles, etc.

Looking for help with ecommerce? Contact the Elastic Path consulting team at [email protected] to learn how our ecommerce strategy and conversion optimization services can improve your business results.