Skip to Main Content

Aug 31, 2011 | 7 minute read

Putting Your Best Site Forward: A/B Testing Webinar Recap

written by Linda Bustos

No matter what the marketing investment, everyone wants to know what kind of ROI to expect. Landing page optimization (or LPO) - just a longer name for conversion testing - is no exception.

For the inaugural Landing Page Optimization Benchmark Report, our friends at MarketingSherpa have compiled research from 2,673 online marketers on landing page optimization’s impact on revenue and its ROI, including which pages and page elements they rate most effective. This information (and more) was shared in our recent webinar: Putting Your Best Site Forward: Making the Case for Consistent Testing on Ecommerce Sites. <--- Replay available on-demand. Let’s explore the juicy stats…

Landing page optimization annualized impact as a % of revenue in 2010

87% of marketers got some kind of result with testing, and close to 20% got double digit results. Percentage lifts mean different things to different organizations – for some, testing gains equate to tens of thousands or even tens of millions of dollars. Also keep in mind profit and loss statements differ across organizations – the key point is the majority of marketers do achieve improvements through testing.

The ROI of landing page optimization and the ability to calculate it

Only 1% of marketers experienced negative or no return on investment. It is interesting that 36% could not or did not calculate ROI, and a further 18% did not know whether testing improved the bottom line.

Keep in mind, this was a survey of various kinds of online marketers, when we look at ecommerce performance, 60% demonstrated positive ROI, and only 27% could not or did not calculate ROI.

The ROI of testing and the ability to calculate it in 2010 by website objective

Also remember not all tests are measuring sales, for example a home page design test that measures a decreased bounce rate. It’s more difficult to attribute a sale to a home page element as there are a number of steps in between that are more influential in “selling.”

Page elements that marketers optimized in 2010

Once again, keep in mind this survey includes lead-gen sites, which may be skewing headline copy, body copy and page header image higher than typical ecommerce (not that ecommerce sites can’t work on these elements, they are just more akin to lead-gen landing pages).

We see here that form logic, location of calls to action, form layout and navigation are all quite important to test, no matter what type of business you’re in.

Pages that rely on sensitive and complex transactions are among the most difficult to optimize:

Difficulty level of optimizing a page or funnel in 2010

Let’s not forget data sources, one of the inputs to an effective conversion testing strategy:

Visitor data types rated “very effective” for relevance, by website objective

Though averages are not predictors of what your individual business can achieve, benchmarks/guidelines can be a helpful reference point, especially when trying to make a case for testing with others in your organization. It's helpful to hear what's working and what isn't across the board in landing page optimization.

In addition to these gems, the webinar discussed:

  • The kind of skills you need to implement a testing program and how to overcome expertise gaps
  • The competing priorities and operational challenges commonly faced by marketers
  • Where to start testing on your online store
  • An ecommerce case study with double-digit success
  • Live optimization of web pages submitted by attendees

Your questions answered

Attendee questions that weren't covered in our limited time slot are answered here by special guest, Boris Grinkot of MarketingSherpa. Additional questions? Fire away in the comments!

What are the best testing tools? How should you pick a conversion testing tool?

If you are just starting out, I would definitely recommend Google's free Website Optimizer, but depending on your needs, you may quickly grow out of it. There are certainly good reasons to pay for other testing tools, but you must discover what your testing bottlenecks are. Starting with a free tool is a great way to get going quickly.

Some testing programs come with algorithms that optimize. Are these looked upon favorably in the industry?

Tools like Maxymiser offer ongoing automated optimization based on visitor behavior. The automation is limited to what test elements you enter -- the software will not write your headlines for you, but it will pick the likely optimal one from a set you provide. The vendor suggests that the software is able to back out (assumed using regression analysis) what content is best suited to a given user based on that user's behavior as compared to other users. I expect that the effectiveness will vary by website, as these tools typically target optimization of content, not of process/functionality. Also, to work well (as with anything statistics-based), they require significant traffic to be effective. As with the promise of multivariate testing, you can only get results if you can supply the tool with a sufficient number of samples. For sites that only get a few thousand visits per month, single-factorial tests are probably more practical.

How important is conversion testing compared to best practices optimization?

Testing is important because it's the only way to know that a "best practice" is any good for your particular site. As very briefly mentioned in the webinar, LPO/CRO is about connecting the objectives of the page (or funnel) with the preferences and motivations of the visitor. Both sides of the equation are unique for each website. So if you are going to use best practices, make sure that they are not based on very broad averages.

It's like some people think that if they use words that have been showing up the highest number of times in retweets, their tweets will be more likely to retweeted. That just doesn't work, and that's a silly misuse of statistics. Use best practices to understand the underlying principles, but adapt them to your objectives and your website visitors. And even within your own site, the objectives of different pages are different, and you may have different traffic sources or other ways of identifying visitor segments having different preferences. We see it in tests all the time that different types of traffic to the same page respond differently to different treatments. As a result, tests often help us create dedicated experiences for each traffic channel, but that's a separate topic for discussion.

I have run several tests but my bottom line does not appear to be moving. Are conversion rates above 4 and 5% really possible? And how can I ensure my tests are impactful?

First of all, there is no "right" conversion rate, or a useful "average" figure to which one should aspire. Looking for a broad average like that just isn't helpful. And certainly conversion rates of over 90% are possible. It all depends on how motivated your visitors are, how optimized your funnel is and what page or process you are actually optimizing.

Also, testing is not a guarantee of bottom line lift. Testing helps you confirm which treatment is the best of the several created. If you are not able to create something that's better than what you already have, testing is just going to confirm that fact for you. Looking at case studies for similar sites might help you generate radically new ideas to test. Also, you may be testing/optimizing the wrong things. Look to your data for the major leaks in the funnel. If you are optimizing a page that has no impact on the bottom line, then optimizing it will not help. You may want to perform a comprehensive sensitivity analysis to understand the impact of incremental conversion rate changes on the pages you are optimizing on your business KPIs.

How do you envision conversion testing changing as we head into 2012? Are there any new trends emerging?

I think the biggest trend in conversion optimization is its increased visibility. And I'm not just saying that because MarketingSherpa just had its inaugural Optimization Summit this year in Atlanta. Conversion Conference East is coming up in NYC in October, and is in 4 locations around the world. Also, Design for Conversion offers a great hands-on format. That's just to name a few. At the same time, data analysis and testing tools are becoming more accessible, while the number of published case studies is growing geometrically. And visibility means it will be on the agenda of corporate leadership, likely meaning that many corporate marketers will find themselves tasked with creating a testing program if they did not already have one in place.

Thank you, Boris!

Looking for help with A/B and multivariate testing? Contact the Elastic Path consulting team at to learn how our conversion optimization services can improve your business results.