Optimization Testing: Should You Avoid Testing During a Traffic Spike?
Many testing gurus wisely advise you to avoid testing near a predictable traffic spike such as the holiday season, major marketing campaign or new product launch. It's easy to misinterpret this advice as "never test during a traffic spike - ever." But this is not the case. What you want to avoid is a change in consumer behavior (traffic volume), motivation and purchase roles (like gift-giver) during your test period that may introduce bias into your experiment.
Testing during a traffic spike has advantages. Higher traffic volume means a shorter test, and your test findings will reflect consumer behavior during your most critical time period. So long as the test begins and completes before significant drop-off occurs, you can benefit from implementing "winning" elements during your most valuable selling period.
But you must test smart.
What to test before launch
As features and value propositions are likely to change, for or any product that will be essentially "replaced" by a new model or version, avoid testing persuasive elements such as:
- Product copy
- Headlines and page titles
- Thumbnail images
- Screenshots and video
- Cross-sell, upsell
- Testimonials, 3rd party reviews
Rather, if you have enough time to begin and complete testing before launch, consider testing elements that are not product-related that may increase conversion across your entire site, including:
- Checkout process
- Point-of-action assurances (privacy policies, security icons and messaging)
- Placement of price, cart button, thumbnail images and other content
- Image sizes
"Enough time" must account for the analysis, hypothesis formation, designs and approvals involved, not just the duration of the test.
Testing during the spike
Your test(s) should begin as soon as traffic starts ramping up so your winning test versions can be rolled out as early as possible. If you have sufficient traffic, you may be able to squeeze in a few rounds of testing - but choose your test design wisely. If you have a lot of important testing ideas, a multivariate test design is more efficient than a series of A/B tests.
Focus your testing on product-specific elements, your product copy, promotions, headlines, etc. Avoid mixing your test with global design elements, it will only drag out your test longer. Your focus is on what helps you sell this individual product.
Don't be discouraged if your test does not show a significant difference between control and treatment versions. It's not a failure - you've learned that the elements you are testing do not impact conversion, and that knowledge is as valuable as knowing something does have an impact!
After the spike
Once traffic has settled and sales velocity slowed, you want to know if consumer behavior is different during a rush vs the rest of the product life cycle. Conduct the same experiments again, and compare results. What you learn from this exercise could be applied to similar products you carry or to future releases of your product.