Last post Chris Goward, author of You Should Test That joined us to answer the tough questions on A/B testing. In his book, Chris also shares a landing page case study from B2B software vendor SAP that's a great example of working within branding constraints that are often imposed on testing programs.
With A/B and multivariate testing, the biggest gains are often achieved by making big changes, like “breaking” page layouts, testing contrasting colors on calls to action, using radically different imagery (or removing images altogether) and slicing-and-dicing form fields. While you can't always get what you want, if you try sometimes, you can get great gains from testing the right stuff
The brand constraints
Strong brands like SAP often come with branding that can limit creativity and flexibility. In this case, pages require a standard, corporate banner that takes up header space and inevitably pushes more content below the fold (wherever that may be for a user). The team had to accommodate fixed column widths, adhere to font and font size standards, work within an approved brand color scheme and select images from an approved list.
Design constraints can be a blessing in disguise, as it keeps your focus on the message of the landing page rather than look and feel. Analysis of the landing page was centered on the clarity of the value proposition, and minimization of distracting design elements that were within the control of the testers given the constraints.
Versions 'A' and 'B' were constructed. 'A' featured a more prominent call-to-action (the Big Orange Button) while minimizing distractions.
'B' baked a conversion form into the landing page, eliminating the additional step on the control and version 'A'.
This test achieved a 32.5% boost in lead generation conversion rate with version 'A', and 17% lift with version 'B'. Perhaps these numbers are as good or better as the results would have been had the team had complete freedom around testing elements.
7 tips for testing within constraints
1. Focus on clarity of the value proposition. Always, always, always start with this, and build it into headlines and body copy.
2. Don't be afraid to flip visuals. When working with rigid templates, moving things from left to right and vice-versa can have surprising impact.
3. If images can’t be changed, try making them bigger (or remove them completely, in the case of the 'hero shot')
4. Don't be afraid to ask for permission to bend the rules. WiderFunnel did win some wiggle room on version 'A' from the branding team on certain elements.
5. Experiment with copy length. This is a great test branding can't touch!
6. Experiment with number of “conversion exits” (calls-to-action within copy).
7. Experiment with button label text and position if color and size can’t be tweaked.
If you're interested in the details of the SAP case, it's published in full in You Should Test That and on Wider Funnel's conversion rate optimization including formation of the hypothesis and technical implementation.