Testing the Untestable: An SEO Title Tag Experiment
Testing title tags: the problems
SERPs (search engine result pages) introduce many uncontrollable variables, such as Google’s penchant for showing different flavors of results, in different areas on the page, for different keyword searches. For example, a search for ‘riding boots’ shows Google Product Search results that can divert a customer from organic listings, star ratings that draw attention to search ads, and brand links to help users narrow their searches.
Notice how the singular “riding boot” and plural “riding boots” show the same page results, but slightly different related Stores results, and omits Product Search results.
If you’re really observant, you’ll notice the related Stores options are different too. Testing is difficult because you have no way of knowing how often these elements are shown or how they may be skewing your results. Not only that, but search positions are always in flux, and vary depending on the exactkeyword phrase queried, the user’s geolocation, whether the user is logged into a Google account (personalized to search history), the “freshness” of results at any given time, and perhaps even involves estimated page load speed and activities of one’s social graph. And did you know that search engines sometimes override your HTML and create their own title for you based on content on your page or what it thinks your page is about? Finally, the biggest hurdle is that even with Google’s own Website Optimizer tool, you cannot tell Google to show one title tag 50% of the time and another the rest. Google will only index your control page. Test versions are served after the referral, if applicable. This makes A/B testing individual pages’ title tags impossible. The only testing you *could* do is in aggregate.
Testing title tags: the workaround
Get Elastic reader Sander Daniels' team at Thumbtack.com found a workaround to A/B testing title tags in aggregate, described in this case study. Using a home-grown testing platform in its ecommerce back end, the test ran for 7 days, Thumbtack split its URLs into 3 buckets, each containing thousands of URLs. The buckets consisted of a control and 2 challenging formats.
Control: Looking for the best [Service] in [City Name]? Challenger A: Get Free Quotes Today From [Service] In [City Name] Challenger B: [Service] in [City Name] – Get Free Quotes Today
To control for the uncontrollables in search, Thumbtack used the ratio of visits to its experimental titles compared to visits to pages in its control group. Rankings were also monitored, and there was no significant change during the test period. The test ran for 7 days, resulting in a 20-30% drop in traffic to its test groups. Traffic rebounded (for the most part) after reverting to the original title tags.
What to test in ecommerce title tags
If you’re bold enough to try a similar experiment (and have the technical judo chops), here are a few hypotheses you could explore:
- Keyword vs. store name at the beginning of the title
- Short vs. long, “keyword stuffed” titles
- Value proposition such as “Free shipping” or “largest selection of” at the beginning of the title
- "Shop" at the beginning of the title to differentiate from informational results such as Wikipedia articles and blog posts
Remember, to make it work:
- You need a fairly large sample size of URLs that you can experiment, and a tool to support it the test
- Your treatment versions must be tested against your control concurrently, not sequentially
- Allow enough clicks to accrue to reach a statistically significant result
- Understand what you are measuring - are search positions affected (for example, if your brand name at the beginning of the title affects your keyword relevance across the board)? Is traffic affected (can you estimate you are getting higher click through)? Are conversions affected (are you attracting the same quality of visitor)?
- And finally, revert title tags back to normal after the test is over to validate your results
Looking for help with ecommerce strategy? Contact the Elastic Path Research & Strategy team at firstname.lastname@example.org to learn how our ecommerce strategy services can improve your business results.