The Definitive Guide To Conversion Optimization

The Definitive Guide To Conversion Optimization

Written by Neil Patel & Joseph Putman

Chapter Four

How To Run Your First A/B Test To Find A Winning Variation

Now that we’ve talked about what conversion rate optimization is, why it’s so valuable, and what kind of data you need to gather to make informed decisions, we’re going to walk you through how to run your first A/B test in order to find a winning combination. This includes what kind of software you need to run a test and how to set up it up.

Selecting The Software

The first decision you need to make is which software you’ll use to carry out your test. There are many testing softwares available, but three of the most popular are Google Content Experiments, Visual Website Optimizer, and Optimizely. Here’s a breakdown of each:


Google Content Experiments is a free service offered through Google Analytics that allows you to test up to five versions of a single page. First, you create a new version of a page, then you plug the new page information into Google Analytics and carry out the test. Content Experiments measures the results, reveals conversion rates for both (or all five), and selects a statistically significant winning variation. The benefit of Google Content Experiments is that it’s free, but the downside is that you have to design and code new variations before testing.

Visual Website Optimizer enables you to carry out and measure tests the same way you can with Google Content Experiments, but it also includes a what-you-see-is-what-you-get (WYSIWYG) editor. The WYSIWYG editor enables you to make changes without having to change the underlying HTML or CSS code to run a test. You can change a headline or button copy and run a test without having to enlist IT resources to set it up. You can also edit the CSS and Javascript if that’s something you know how to do, but it’s not required for many tests. Thus, the benefit of VWO and many other testing platforms is that they enable you to run tests without using IT resources, something that is in short supply for most businesses.

Optimizely is similar to VWO in that it also provides a WYSIWYG editor. You can make changes and run tests without the need to edit the underlying code of the page. Both tools are very similar and offer nearly the same features.

This is a screenshot of what test results look like in Optimizely.

In the end, the testing software you use will largely come down to preference. If you don’t mind editing pages before testing and don’t want to pay a monthly subscription fee, then you can use Google Content Experiments. But if running tests without enlisting IT is important to you, then Visual Website Optimizer, Optimizely, or one of the other testing options is a better choice.

The most important point is that you have a software that makes it easy to run your tests and measure the results. All of these tools automatically calculate the conversion rates and measure the statistical likelihood that one variation performs better than another, which leads us to an important point.

It’s recommended to run all of your tests until there is at least a 95% chance to beat the original. This number will fluctuate throughout the test, and the smaller the difference in conversions, the longer it will take to reach a 95% or higher likelihood. However, by waiting this long, you increase the chances that the increase (or decrease) in conversions is statistically relevant.

In addition, it’s also considered a best practice to run a test for at least seven days. Sometimes this isn’t necessary, depending on the level of traffic your site has, but results can fluctuate, and sometimes a seemingly winning variation won’t last all seven days. And if it can’t last seven days, do you really want to implement that result to be used on your website? Probably not.

So for testing,
there are two important rules of thumb to follow:

  1. Always test until there is at least a 95% chance to beat the original variation (or visa versa),


  2. Consider running your test for 7 days to make sure the measured result can stand the test of time.

Coming Up With A List Of Tests

As mentioned, in order to set up your initial test, you first need to pick the software you’ll use. For the sake of our example, we’ll use Visual Website Optimizer, which means step one is taken care of.

Step two is to decide what to test, but before we do, we have to figure out what the goals are for your site. Are you attempting to generate more leads for your service business or more sign ups for your SaaS business? Knowing your goals determines what tests to run.

We’d also like to remind you that the most important goal for your business is what needs to be measured. If your current goal is to generate more free trial sign ups, then you need to use that as your primary conversion goal, and if your goal is more paid account sign ups, then that’s what you should measure on your site. Every test should improve conversions for the most important goal on your site.

For the sake of our example, we’re going to run a hypothetical test on, and we’re going to measure free trial sign ups as our goal. This means the number of people who sign up for a free trial is the metric we’re going to use to determine success.

To begin we need to come up with a list of tests we can run. This normally would be based on data we’ve gathered, but for the sake of the example, we’ll just make a list of the standard tests that make sense for most sites: headline, sub-headlines, and call-to-action button copy. All of these items have been shown to impact conversions on other sites, so we’ll come up with new variations to test on Crazy Egg.

Prioritizing The List

At this point you’ll usually have a short to semi-long list of tests you can run. It’s important to prioritize the tests based on which one you think will have the biggest impact on your conversions. Do you think adding a testimonial is more likely to lead to a win, or do you think new call-to-action button copy will make the difference? Either way, the goal is to come up with a list of tests you can run, and then to prioritize them based on which one you think will make the biggest difference.

It’s also good to keep track of the test ideas that you have. You may want to test a new headline, try new call-to-action button copy, and consider adding a free shipping icon in a prominent place on your site. These are all great ideas, but if you test them one at a time, you’ll need to create a list so you can keep track of your ideas and not lose any really good ones. The best way to do this is by creating a spreadsheet where you can log and prioritize your ideas for future tests.

This is an example of a spreadsheet you can create to keep track and prioritize your testing ideas.

Something else to keep in mind is that testing one item at a time can lead to incremental improvements, but big changes are more likely to lead to a big win. For example, testing ten small elements may improve conversions by 25%, but one big change can cause them to go up by 125%. It’s recommended that you experiment with bigger changes in order to get bigger wins, but when first getting started, small tests are good to run as a way to get your feet wet and to learn how to run and measure tests.

So in order to start small and get our feet wet, we’re going to make a short list of tests we can run, prioritize them, and then decide which one to run first. Here’s the short list (which we’ll pretend is based on data we’ve gathered prior to coming up with the list):

  1. A new headline: Crazy Egg Reveals More Than Regular Analytics
  2. A new description: Use Crazy Egg heat maps to find insights on your site Google Analytics can only dream about.
  3. New button copy: Start Optimizing Today

It’s difficult to know which one will improve conversion rates or which one will improve them the most, but our best guess is that the headline will make a big difference, so we’ll start there.

Setting Up The Test

The next step is to set up the test. For our experiment we’re going to use Visual Website Optimizer as our testing platform.

First, we’re going to log in to VWO and click “Create Test” to get started.

Next, we’re going to select “A/B test” in order to set up an A/B split test. You can always run a multivariate test or a split URL test if that makes more sense for the test you’re running, but we recommend starting out with A/B tests until you get more comfortable with running an experiment and evaluating the results.

The next step is to enter the url we’ll be testing.

Next, we’ll use the VWO WYSIWYG editor to create a variation with the new headline. As you’ll notice, it’s really easy to change the headline, and the font and size are the same as the current version of the site. You can use html to customize the new version, but simple tests can be run without needing to write a single line of code.

Now that we’re on step four, we’ll enter the actions we would like to track. The first step we’d like to measure for this test is the number of people that go from the homepage to the next step in the funnel which is After that, we’d like to track the number of people that sign up for a free account which means they land on this page:

It’s important to remember that the final conversion goal is what really matters. It’s great to get more visitors to click from the homepage to the signup page, but what matters even more is the number of people that end up signing up for a free trial. And then after that, what matters the most is how many people sign up for a paid subscription. Those are the conversion goals that matter the most for a SaaS business, not the number of people who click through from the homepage to the signup page.

With that said, it’s useful to know what the conversion rates are for each step of the process. Sometimes you may notice that there’s a bottleneck between the homepage and the signup page, so you come up with a test that you hope will get more people to click through. But if you do, remember to always test for the final conversion since just because more people click through doesn’t mean more people will sign up for your service or for a free trial. And just because more people sign up for a free trial doesn’t mean more will become paid customers. You need to measure each step of the process, and you should always remember to choose the winner based on the results that will have the biggest impact on your bottom line.

Now finally, for the last step of the process, we’ll name the test, add some notes, determine what percentage of traffic to use, and decide whether or not to filter out any types of traffic such as location, paid, organic, etc. Here’s a further description of each:

  • The name is simply what you’d like to call the test so you can differentiate it from different tests you’ve run. A simple name like “Crazy Egg Homepage Headline” works just fine.

  • Notes are where you can keep track of the hypothesis for the test or any details about the different variations. This part is optional and can be used if you’d like to record notes about the experiment you’re running.

  • The percentage of traffic gives you the option of not testing on all of your traffic. Most of the time you’ll leave it at 100%, but if needed for whatever reason, you can always lower it so not every visitor is part of the experiment.

  • Target test to a segment allows you to filter out certain kinds of traffic. You could filter out mobile traffic, desktop traffic, or international visitors, depending on what makes sense for your business. For example, if you run a U.S. based site that only sells to customers in the U.S., you can filter out international traffic so it doesn’t impact the results in any way.

Your new variation is now ready for testing. Now that it’s created, VWO will automatically show the new version to roughly half of your traffic and the original version (the control) to the other half. After enough traffic has been sent to both versions, a winning variation will be selected which you can then implement permanently (or at least until the next winning variation is found). If the new version does not improve conversion rates, then you can stop the test and move on to the next option in your list of conversion tests for your site.

Coming up in the next chapter we’ll talk about how to interpret the results in order to choose winning variations.

Chapter Four Summary

  • Before carrying out a test you first need to select the testing software that you’ll use. Three popular options are Google Content Experiments, Visual Website Optimizer, and Optimizely.
  • Google Content Experiments is a free tool offered by Google, but it requires you to design, code, and implement a new variation before testing.
  • The benefit of tools like Visual Website Optimizer and Optimizely is that they enable you to make changes and run tests without fully designing and coding a new version for testing. Sometimes it’s necessary to do extra coding, but often you can run simple tests without needing any IT resources at all.
  • The biggest benefit of using testing software is that it automatically calculates the conversion rate and likelihood that a new test will be a winning version. All of this can be measured on your own, but testing platforms make it much easier to measure and determine a winning variation.

Four ways to share