Optimization Testing: Should You Avoid Testing During a Traffic Spike?

In our most recent webinar Taking Your Site Performance to The Next Level With Optimization Testing, an audience member asked a great question about site testing near a product launch. A perfect industry example is packaged software. A software product may release a new version every year or two. For the first few months after release, marketing runs a number of campaigns, traffic volume spikes and sales go through the roof. For the rest of the version's life cycle, sales are made less frequently, and the site attracts a mix of new customers and existing customers looking for product information and support. Should the ecommerce team conduct optimization testing in the months before the new release?

Many testing gurus wisely advise you to avoid testing near a predictable traffic spike such as the holiday season, major marketing campaign or new product launch. It's easy to misinterpret this advice as "never test during a traffic spike - ever." But this is not the case. What you want to avoid is a change in consumer behavior (traffic volume), motivation and purchase roles (like gift-giver) during your test period that may introduce bias into your experiment.

Testing during a traffic spike has advantages. Higher traffic volume means a shorter test, and your test findings will reflect consumer behavior during your most critical time period. So long as the test begins and completes before significant drop-off occurs, you can benefit from implementing "winning" elements during your most valuable selling period.

But you must test smart.

What to test before launch

As features and value propositions are likely to change, for or any product that will be essentially "replaced" by a new model or version, avoid testing persuasive elements such as:

• Product copy
• Headlines and page titles
• Thumbnail images
• Screenshots and video
• Promotions
• Cross-sell, upsell
• Testimonials, 3rd party reviews
• Pricing

Rather, if you have enough time to begin and complete testing before launch, consider testing elements that are not product-related that may increase conversion across your entire site, including:

• Checkout process
• Point-of-action assurances (privacy policies, security icons and messaging)
• Placement of price, cart button, thumbnail images and other content
• Image sizes

"Enough time" must account for the analysis, hypothesis formation, designs and approvals involved, not just the duration of the test.

Testing during the spike

Your test(s) should begin as soon as traffic starts ramping up so your winning test versions can be rolled out as early as possible. If you have sufficient traffic, you may be able to squeeze in a few rounds of testing - but choose your test design wisely. If you have a lot of important testing ideas, a multivariate test design is more efficient than a series of A/B tests.

Focus your testing on product-specific elements, your product copy, promotions, headlines, etc. Avoid mixing your test with global design elements, it will only drag out your test longer. Your focus is on what helps you sell this individual product.

Don't be discouraged if your test does not show a significant difference between control and treatment versions. It's not a failure - you've learned that the elements you are testing do not impact conversion, and that knowledge is as valuable as knowing something does have an impact!

After the spike

Once traffic has settled and sales velocity slowed, you want to know if consumer behavior is different during a rush vs the rest of the product life cycle. Conduct the same experiments again, and compare results. What you learn from this exercise could be applied to similar products you carry or to future releases of your product.