What is A/B Testing? Examples, Tools & How to do A/B Testing?

Have you ever run into this situation?

You have done all the work. You’ve built a new webpage or email campaign, and you are ready to release it into the wild. Then, right at the last minute, you begin guessing yourself.

Maybe the image on your webpage isn’t quite right? Should your email headline be punchier?

When marketing a business, you need to make tons of decisions, even the tiny ones. How can you be sure that the choices you’re making are the right ones?

That’s where A/B testing comes in handy!

In this guide, you’ll learn everything about A/B testing, including its definition, importance, challenges, and steps to running.

In order not to waste your time, we should begin now!

Table Of Contents

• What is A/B testing?
• How does A/B testing work?
• Why is A/B testing essential?
• 3 A/B testing practical examples
  • Example 1. Groove’s landing page redesign
  • Example 2. Performable’s CTA button color change
  • Example 3. Humana’s banner test
• 7 Steps to running A/B testing
  • #1. Choose what you want to test
  • #2. Identify your goal
  • #3. Set up a “control” and a “challenger”
  • #4. Determine your sample size (if applicable)
  • #5. Test both variations at the same time
  • #6. Take action based on your results
  • #7. Plan your next A/B testing
• 3 common A/B testing tools
  • 1. Optimizely
  • 2. VWO
  • 3. HubSpot
• The bottom line

What is A/B testing?

A/B testing (or split testing) is the act of creating two versions of your marketing material (including option A and option B), then releasing them both to see which one performs better for a given conversion goal.

Marketers can use A/B testing to test a variety of marketing elements, such as copy, navigation, design/ visual elements, submission forms, and calls to action (CTAs).

So, no matter if you’re trying to boost the open rate with the best headline, or looking for the best image to place on your webpage, running A/B testing is an excellent tactic to decipher the version with the best conversion optimization strategy.

A/B testing not only gives you a better understanding as to which strategies work best, but the data and analytics gathered will ultimately provide valuable insights into your business that will benefit future marketing campaigns.

The concept of A/B tests is quite similar to the scientific method. If you want to know what happens when you change one thing, you need to set up a situation where only that one thing changes.

Remember any experiments you conducted in elementary school? If you put two seeds in two cups of dirt, then place one in the dark, and the other by the window, you’ll see different results. This kind of experimental setup is similar to A/B testing.

How does A/B testing work?

A/B testing is performed by taking a marketing element (i.e., image, copy, layout, email subject line) and creating a second version of this piece. The second version will contain minor modifications. Think simple changes such as a slightly varied copy or a different CTA button.

Differences between the two versions should be minimal, so you can isolate what exactly is affecting engagement from your audience. Then, you will show these two versions to two similarly sized audiences and analyze which one performed better over a particular period (long enough to make accurate concussions about your results).

Half of your traffic will see the original version of the page (called the control), and the other half will see the other version (called the variation).

As your visitors are served either the control or variation, their engagement with each experience is measured and collected in an analytics dashboard, then analyzed through a statiscal engine. You can determine whether changing the experience had a positive, negative, or no effect on their behavior.

Why is A/B testing essential?

Running A/B testing can be a great way to learn how to drive traffic to your website effectively and generate more leads from the visits you’re getting. Even minor tweaks to a landing page, email, or calls-to-action can affect the number of leads your company attracts and converts.

The potential advantages of A/B testing include:

• Improved content
• Higher engagement
• Higher conversion rates
• Reduced bounces rates on pages

When it comes to customer-facing content, there’s so much you can evaluate with A/B testing. Common targets often include:

• Paid advertising
• Email campaigns
• Multimedia marketing strategies
• Newsletters
• Website design

In each category, you’re able to conduct A/B testing on any number of variables. For example, if you are testing your site’s design, you can try different options such as:

• Layout
• Images
• Color scheme
• Headings and subheadings
• Product pricing
• Special offers
• Call-to-action button

Almost any element of your content can actually be tested on, as long as you’re establishing a clearly defined control and variable. This can open up a realm of countless optimization opportunities as you use insight collected from user behavior to inform your design and content creation decision moving forward.

3 A/B testing practical examples

You might wonder whether A/B testing really works or not. And if it works, are there any live examples you can get inspiration from?

We can say that A/B testing works. And yes, there are many practical examples you can take a cue from. Let’s look at 3 A/B testing examples right below!

Example 1. Groove’s landing page redesign

Every marketer will need to build a landing page at some point. Nevertheless, creating a landing page that converts is challenging.

Groove experienced that first hand when the company learned one of its landing pages was only converting at 2.3%.

To make matters worse, Groove was unsure why the page wasn’t converting. So, its team went on a journey to figure it out. They looked up different resources and talked to marketing experts to find the problem. They also reached out to real users and asked for their opinions.

That’s when Groove learned that the messaging was all wrong. They decided to rebuild their landing page, focused on the copy and design. Only when the copy was completely rewritten and approved, did they start the visual aspect of designing.

Ultimately, the tweaks to messaging doubled their conversions to 4.3%. It may not look like much, but it was definitely worth it.

Here’s how Groove’s homepage looked like before the A/B testing and how it looked afterward.

Example 2. Performable’s CTA button color change

Colors can have a profound effect on your visitor’s response to your site.

Well, if you don’t believe that, here is a case study that will level your doubts.

Performable, a marketing automation company, desired to increase its conversion rate significantly. Of all the possibilities they’d have tried to achieve this goal, they simply changed the color of their CTA button.

By changing the color of the main CTA button from green to red, the company increased clicks by 21%.

According to HubSpot’s analysis, the color didn’t matter as much as the contrast. The green in Perfomable’s logo meant the CTA button was dampened and got blended into the background.

By changing it to red - green’s perfect contrast - Performable made it stand out and motivate visitors to click.

Example 3. Humana’s banner test

Many companies showcase large banners at the top of the page. However, if the banner isn’t optimal, it could end up doing more harm than good.

That’s why Humana, a healthcare insurance provider, wanted to test its landing page banners.

In the control, the company used a banner that displayed a full paragraph of copy, a weak CTA, and no clear and concise message.

Nevertheless, for variation B, Humana decided to simplify the message with a strong and prominent CTA button.

As a result, the variation ended up achieving 433% more click-throughs than the control.

The A/B testing examples listed above give you an idea of the results you can achieve through testing.

The question right now is: How can you implement your own A/B testing to make it successful?

The answer will be displayed in the next section, with 7 steps to running an effective A/B test.

Let’s explore!

7 Steps to running A/B testing

#1. Choose what you want to test

As you optimize your webpage and email campaigns, you might find many variables you want to test.

However, to measure how effective a change is, you should isolate one “independent variable” and evaluate its performance. Otherwise, you cannot be sure which one was responsible for changes in performance.

For example, if you are trying to generate more organic traffic, test an element that impacts SEO, like blog post length. Or, to boost your conversion rate, you might begin with the headline or CTA buttons.

Keep in mind that even simple changes, such as changing the words on your CTA button or the image in your email, can drive significant improvements (you can see obviously from the above examples!). Actually, these sorts of changes are often easier to measure than the bigger ones.

#2. Identify your goal

Although you want to measure a number of metrics, narrow them down to 1 or 2, at least to start. Do it before you set up the second variation. This is called your “dependent variable.”

You might state a hypothesis and examine your results based on this prediction. For example, you notice that the CTA taking people to your online store is tucked away at the bottom of the email. You suspect that if you move it to the top, you can more effectively encourage people to visit your site.

If you wait until the last minute to think about which metrics are vital to you, what your specific goals are, and how the changes you’re proposing might affect user behavior, then you might not set up the A/B test in the most effective way.

#3. Set up a “control” and a “challenger”

Now you have your independent variable, dependent variable, as well as your desired outcome.

Use these pieces of information to set up the unaltered version of whatever you are testing as your “control.” If you are testing a webpage, it’s the unaltered webpage as it exists already. If you are testing a landing page, it would be the landing page copy and design you’d normally use.

Then, build another variation, or a ‘challenger” - the landing page, website, or email you’ll test against your control. For instance, if you wonder whether including a testimonial on a landing page would make a difference, set up your control page with no testimonials. And then, create your challenger with a testimonial.

#4. Determine your sample size (if applicable)

How you determine your sample size depends on your A/B testing tool, as well as the type of A/B testing you are running.

If you are testing an email, for example, you might want to send an A/B test to a smaller portion of your list to get statistically significant results. Ultimately, you can pick a winner and send the winning variation to the rest of the list.

In case you are testing something that doesn’t have a finite audience, like a webpage, how long you keep your A/B test running will directly affect your sample size.

You should keep your test running long enough to obtain a substantial number of views; otherwise, it will be hard to tell whether there was a statistically significant difference between the two variations.

#5. Test both variations at the same time

If you planned to run Version A during one month and Version B a month later, how could you know whether the performance change was caused by the different copy or the different month?

That’s why when you run A/B testing, you need to implement the two variations simultaneously; otherwise, you may be left second-guessing your final results.

The only exception is you are testing timing itself, such as finding the optimal times for sending out emails. This is an interesting thing to test, because depending on what your company offers and who your audience is, the optimal time for the engagement can vary significantly by industry and target market.

#6. Take action based on your results

In case one variation is statistically better than the other, you’ve got a winner. Complete your test by disabling the losing one.

If neither of the variations is statistically better, chances are the variable you tested didn’t impact results, and you will need to mark the test as inconclusive. In this case, you can stick with the original version, or run another test. Remember to take advantage of the failed data to figure out a new iteration on your new test.

While A/B testing helps you impact results from case to case, you can also apply the lessons you learn from each test and apply it to future efforts.

For instance, if you have conducted A/B tests in your email marketing and found that using numbers in email headlines generates better clickthrough rates, you might want to consider using that tactic in more of your emails.

#7. Plan your next A/B testing

The A/B test you’ve done may help you discover a new way to make your marketing efforts more effective.

However, don’t stop there. Keep in mind that there’s always room for more optimization.

You can try carrying out an A/B test on another feature of the email or webpage you just did a test on. Let’s say, if you just tested a headline on a heading page, why not do a new test on CTA? Or images? Or color scheme?

Always keep an eye out for opportunities to boost your conversion rates and leads.

3 common A/B testing tools

To help you implement A/B tests smoothly, and most importantly, help you find the answers to your own specific situation, we’ve curated 3 of the best A/B testing tools.

1. Optimizely

With 24 Fortune 100 companies as customers, Optimizely is actually a big kid on campus. It’s a digital experimentation platform aimed at enterprise customers exclusively.

With its powerful A/B and multi-page experimentation tool, you’re able to run multiple experiments on one page simultaneously. This allows you to test different variables of your webpage.

The platform also provides testing on dynamic websites, various experiment dimensions, such as ad campaign, geography, cookies, and various experiment segmentation parameters like browser, device, and campaign. You can also use Optimizely beyond your website, expand to mobile apps, messaging platforms, and much more, so you can optimize your entire customer experience.

Pricing plans: Plans at Optimizely are custom-made by their sales team to match your specific needs.

2. VWO

VWO (Visual Website Optimizer) is incredibly popular in the marketing space. In addition to serving as a top choice for companies with smaller budgets, it’s also frequently used in conjunction with Optimizely by businesses who run complicated testing campaigns.

You can get an extensive testing suite of tools for A/B, multivariate, and split URL experiments. Plus, the platform offers a visual editor for building variations without any code, as well as advanced targeting and segmentation options. Its behavioral segmentation lets you target audiences in a way that a lot of more expensive tools don’t.

To gauge your tests’ performance, VWO offers a robust reporting dashboard. It also provides a SmartStats feature that leverages Bayesian statistics to help you run tests faster, give you more control of your tests, and reach more accurate conclusions.

Pricing plan: VWO Testing Plan starts from $199/ month

3. HubSpot

In fact, HubSpot isn’t the platform for you if you only want an A/B testing tool. It is a full suite of marketing software divided into four key products.

Firstly, there is a CRM (Customer Relationship Management) tool for handling customer data, and then you have Marketing Hub, which includes a number of tools for creating content, landing pages, managing leads, and running A/B testing - among other things.

The platform also offers its Sales Hub with a collection of specialist tools for improving and automating your sales processes, together with the Service Hub, which helps your team provide outstanding customer service.

To unlock its A/B testing features, you need to sign up to the Professional Plan of its Marketing Hub. This costs you a minimum of $800 per month for 1,000 CRM contacts, along with a $3,000 onboarding fee and an extra $50 per month for each additional 1,000 CRM contacts.

All of its software is industry-standard, so you don’t need to worry about quality. Although HubSpot tends to work out as an expensive option, you’ll get a lot of tools built into a single system. That means you don’t have to spend a lot of time moving between different software.

HubSpot has recently collaborated with Kissmetrics to launch an A/B testing kit for free. This kit includes an A/B test tracking template, a how-to guide, and a statistical significance calculator. You can read it for more information.

The bottom line

A/B testing is undoubtedly an efficient way to boost your business performance. With A/B testing, you don’t have to wonder how a small change could impact your click-through rates, open rates, revenue, and other vital metrics. You can also see for yourself and have confidence that your campaigns and materials are optimized for the best results.

Start thinking about which small tweaks you would like to test out in your emails, landing pages, paid ad campaigns, and more. Try A/B testing, and then let the results speak for themselves.