While it's true that our guts often get it wrong and testing can help that, they also influence testing. Most tests are set up based on a gut feeling that changing XYZ will cause a lift. We may have an "informed opinion" based on research, best practices or case study references, but some of our beliefs are shaped by the following 5 factors:
In general, guys and gals tend to shop differently, both offline and online. While user experience demands are quite similar between the sexes, save a man's "need for speed," men are typically "buyers" (get in and get out) and women are "shoppers" (open to browsing or "window shopping" online). Men prefer coupons, and women prefer sales. Women and men demonstrate different navigation and color preferences and may respond differently to your site's design. They respond to images and even humor differently, so it's likely when we ourselves are making design decisions, we may be biased by our own biology. Smart testers challenge what appeals to themselves vs. what might appeal to an opposite, whether by gender or another characteristic.
2. Personality Type
If you're familiar with the 4 buying modalities, you understand why some people respond more to impulse offers (the Spontaneous), others want the latest model (the Competitive), some will read every product review, starting with the negative ones (the Methodical), and others want the human touch in their buying experience (the Humanistic).
If you ration that "When I shop, I want to do X, Y or Z, therefore our website should be like this," you may make design and content decisions that don't speak to most of your customers. Persona development can help you think like a customer, and test with real customers that are different than you to understand where the gaps in your user experience are.
3. Level of Tech Savvy
If you're reading Get Elastic, chances are you work on the Web, and using the Web comes naturally. It's easy to forget that some folks don't know how to sort search results, may not understand how to use a promo code or the difference between log in and sign up. It boggles our minds how conversion could drop when a website redesign fixes glaring usability problems and gets a significant aesthetic upgrade. For this reason, our technorati guts often get it wrong. We need the help of the average 'Net Joe. This is especially true with mobile experiences.
4. Your Industry Experience
Being an expert in your business can make you very far removed from your customers. Your information architecture may make sense to everyone internally, but not to prospects (especially if your product is unique and innovative). Not everyone understands your terminology or knows how to navigate your website intuitively. If you're very passionate about your products, you may (erroneously) believe every site visitor is going to devour your web copy page by page, word for word, and wonder why conversion rates are stuck when there's so much information about the benefits of your goods.
Steve Krug's legendary book Don't Make Me Think illustrates this perfectly:
Usability studies with prospects unfamiliar to your company can be quite eye-opening.
5. Your Role in Your Organization
Once again, Don't Make Me Think nails it. Your opinion of the website may depend on what role you play in the company. Branding and marketing folks may have different ideas from UX leads and designers. Or C-level executives may have different views of the web site from the web sales team.
What matters is how your customer feels about your site. Don't make decisions in absence of customer testing / feedback.
Is Gut Feel All Bad?
Despite its shortcomings, we can embrace our gut feel. We should apply experience and intuition to web decisions -- including how we design our A/B and multivariate tests. But it's important that we recognize our own bias and how it influences the decisions we make, so we can design experiments that challenge our own beliefs to truly learn what works and does not on our websites.