Sign In

Internet Performance Delivered right to your inbox

Myth: Father Knows Best. It’s Time to A/B Test.

Father Knows Best was an American comedy that ran on radio and then television from 1949 to 1960. The character of Jim (Father) has strong influence over his family with the best of intentions. I mean, just look at this guy. He radiates confidence and authority. With a character like Jim, it’s easy to forget the most important, invisible star that made this show a long running smash hit: The Audience.

Likewise, businesses with talented, well-meaning employees may also forget the unseen star of the company’s online success:
The User.

Quite unintentionally, our managers, peers and expertise can become the stars of the website. We may even forget to question how the changes we make can impact our audience. Let’s face it. We can all be like Jim, when we think we know what’s best for the company website without really knowing. For example: A company might think adding a section of their employees may make the business appear too small for people to be likely to engage. On the other hand, a user may appreciate this transparency, feel more connected, trusting and be more willing to engage. There is a way to uncover which scenario works better with quantitative data to back it up. It’s called A/B testing.

So, what is A/B testing?

An A/B test simply measures the impact of a change (big or small) to your current website.

To A/B test a web page, create a copy of the original page (control) and add the change you want to test to the new page (variation). The change may be a different content heading, a layout adjustment, an alternate image, CTA button, font, color, etc.

A specified portion of users will see the control and the rest will see the variation without being aware that there are two versions. One version may influence a user’s behavior resulting in a measurable outcome such as a significant revenue gain, increased form submissions, etc.

Google Experiments or other A/B testing software can be used to set up the experiment goals and collect data on both versions for a period of time long enough to determine a winner or learn that there is none. And most importantly, it brings us valuable insight into what our users prefer. Over time and testing, we can get to know our users better and make informed decisions when it comes to change.


It’s good practice to limit the number of changes to one per test. That way, you can be certain about what made the impact.

Test Before Change

The larger your company, the more departments it has. People in different departments have different perspectives and priorities. And nearly everyone has opinions about the company website. Just ask. Input and suggestions for site changes can vary greatly or even conflict with other requests and points of view. The more requests for change, the more challenging it can get to know what to prioritize, implement or say no to. Your site may become muddled over time if you say yes to all requests without vetting them, especially if they’re from vastly different sources. Each person has unique value and expertise. However, with A/B testing, we’re able to find out how these requests resonate with users before implementing them across our site.


When you hear ideas about changing the site, it’s the perfect opportunity to suggest A/B testing and have the requester fill out an A/B test request form. All ideas to be potentially tested should be well thought out with a good hypothesis. Otherwise it’s just throwing spaghetti against the wall to see what sticks.

Sample A/B Test Request Form

Who is the test targeted towards?
Segment examples: All users, mobile users, new users, returning users, German users, etc.

What specifically do you want to test?
Experiment examples: an image, a CTA, content, a color, page layout, etc. – one change per test please!

What do you hope your test achieves using a specific test metric to measure the results?
Goal examples: Increase revenue by 5%, decrease bounce rate by 10%, increase form submissions by 15%, etc.

What is the page you want to test?
The specific URL goes here. Testers, make sure this test doesn’t interfere with any other A/B test, promotion, holiday, campaign, etc. on your site.

Why do you want to test this?
Is your test idea based on a successful case study? Does it come from a knowledgeable resource? Or does it arise from a known issue or is it a gut feeling? Please be specific.

Start an A/B Testing Culture

So how do you begin to spread awareness within your company about the importance of the user’s viewpoint and A/B testing for your site?

  1. Bring up A/B testing in meetings as a means to test new ideas or resolve internal conflicts.
  2. Create an outlet for employees to share suggestions with you for future A/B tests, using tools such a the “A/B Test Request Form” (detailed above).
  3. Share any interesting A/B test results (both wins and losses) with your company.
  4. Share A/B testing case studies and resources with your colleagues like this:whichtestwon.com
    The folks at WhichTestWon post weekly, A/B tests results from companies, right after you see both versions and guess yourself. I guessed wrong so many times, it dispelled some false beliefs I had about what works on a website. I knew best only about 50% of the time according to the results. Lesson learned: If you’ve never tested it before, don’t assume how your users will respond.

Challenge

Take your best guess at Which Test Won for 10 weeks. WhichTestWon presents a different A/B test every week. Check out both versions and make your best guess before looking at the results. Keep track of how many times you guessed right and wrong. It may surprise you, annoy you, or delight you. But best of all, it will make you aware that sometimes your instincts are wrong and you may not know what’s best for the user despite your opinion or experience!

A/B See

I recently attended and highly recommend this conference for anyone who wants to begin or refine A/B testing: WhichTestWon: The Live Event – the most inspirational and useful conference on A/B and multivariate testing.

When it comes to change on a website, it’s really all about the users. Popular opinions, good intentions, years of experience, raw talent, position on the food chain and great ideas don’t trump data. A/B testing provides solid proof of what does & doesn’t work on your website. When you A/B test you move beyond hoping and guessing, into data-driven, informed design. Let the data dictate change. Remember, Dad makes mistakes just like everyone else!


Share Now

Whois: Lynda Straffin

Lynda Straffin is a front-end web developer for Dyn, the world leader in Internet Performance Solutions that delivers traffic management, message management, and performance assurance. You can follow her creative endeavors on Twitter.