7 A/B Testing Blunders That Even Experts Make

ab testing mistakes

You already know about A/B testing,but have you ever run an A/B test? Sadly, chances are you haven’t. Although conversion optimization has been around for years, it didn’t start getting popular until the last few years.

Why?

It’s because people are talking about the results they are experiencing from conversion optimization. Before you go out there and start running A/B tests in hopes to get the same results for your business, however, there are a few things you need to know.

Here are 7 A/B testing blunders you need to avoid:

Blunder #1: Believing what you read

Everyone talks about A/B testing results and how one simple change really boosted their conversions. When you see these results, you are likely to want to copy the same tests on your own website. But a lot of people run A/B tests incorrectly, which is why you need to learn to read between the lines.

red and green button

 

For example, if you read about this A/B test from Hubspot, it seems that having a red call-to-action button outperforms a green call-to-action button. But if you read carefully, you’ll notice that they mention that the red button had a 21% increase in click-throughs over the green button. That doesn’t mean that they saw a 21% increase in conversion rates.

Just because someone clicks a button doesn’t mean they are more likely to convert than if there were following a different variation. Now, the article by Hubspot does mention how they also saw a “21% increase at the bottom”, but it still doesn’t clearly state if they got an increase in conversions by 21%.

By no means am I saying the article by Hubspot isn’t useful, but more so I am trying to point out that you need to be careful about how you interpret the data. Don’t just assume that if someone ran a test and called it a “winning variation” that it is a winning variation. And, most importantly, don’t expect to take that same test, implement it on your website and achieve similar results.

Blunder #2: Ending tests when they are statistically significant

If an A/B testing software shows the results as statistically significant, you should end the test, right? In most cases that would be true, but if you don’t have enough conversions on each variable, you shouldn’t stop the test.

My rule of thumb is each variation should have at least 100 conversions and the test should run for at least two weeks.

When I started running A/B tests on Crazy Egg three years ago, I ran around seven tests in five months. The A/B testing software showed that we boosted our conversions by 41%. But we didn’t see any increase in our revenue.

Why? Because we didn’t run each test long enough. Stopping a test when the winning variation had forty-one conversions and the losing version had fifteen conversions was a bad idea because things can change quickly…especially if the test has been only running for a few days.

Those results didn’t hold true in the long run, which is why we didn’t see revenue increases. Make sure you run your tests long enough.

Blunder #3: Expecting big wins from small changes

If small changes are providing huge gains, something else is wrong with your design or copy. The conversion wins small changes provide typically don’t hold.

The biggest conversion boosts are going to come from drastic changes. So if you really want to move your conversion rates, don’t focus on small changes. Instead, focus on drastic changes as they are the ones that boost your revenue.

When you are starting out, you could try small tweaks to your design and copy to see if your conversion rates increase, but eventually you’ll need to focus on the big wins.

What I like doing is to focus on the drastic changes. Once I feel like I’ve maximized their potential, I then focus on the small changes.

crazyegg homepage

A good example of this was when we first got huge wins with Crazy Egg by changing the homepage to a long sales letter. After we made that drastic change, we then tested call-to-actions, button colors and other small things like that.

Blunder #4: The first step in A/B testing is to come up with a test

If you want to dive right in and start testing variations of your web page, that’s fine. I hope things work out for you, but chances are you will lose some money.

You aren’t the one buying your own product or service; it’s other people. So why should you base your A/B tests on what you think people will want to see?

Instead, you should start off by surveying your visitors. Ask them questions like:

  • What else would you like to see on this page?
  • What can we help you with?
  • Why didn’t you complete the purchase? (great question to use on your checkout page if someone has been idle for more than 30 seconds)
  • What could have we done to convince you to complete the purchase?

Getting qualitative data from your users will help you pinpoint what’s wrong with your messaging. You can then take that data to make changes and test them out to see if you can find a winning variation.

But before you start the test, you need to create an A/A test in which you test the current version of your website against itself. You’re doing this to test out the accuracy of the A/B testing software you are using and the accuracy of the data.

Once the A/A test looks good, you can start your first A/B test.

Blunder #5: Running a lot of tests on a regular basis

Just the other day, I sat down with an entrepreneur who claimed his company were experts in A/B testing because they run over twenty A/B tests each month.

When I started to dig a bit deeper, I found out that their website had a lot of traffic, which is why they were able to go through so many tests so fast. In addition, they didn’t have a ton of pages, so it was easy for their design team to make changes.

Upon hearing all of the evidence, I explained to the person why it’s bad to run so many tests in a short period of time.

For starters, they weren’t basing their tests off existing data, and they weren’t running them long enough. For this reason they weren’t seeing big revenue increases.

To make matters worse, they had a lot of losing tests, which was causing them to lose a lot of money. A temporary decrease in conversion rates means you temporary lose money.

If you can collect enough data, create tests based on the data and then learn from the tests, that’s fine. But it’s unrealistic to do all of that in a very short period of time.

Don’t focus on quantity; focus on quality. Always run tests based on data and learn throughout the process… even if that slows down your A/B testing schedule.

The biggest thing you need to take away from this blunder is how important it is to learn from your tests. Learning takes time, so don’t try to force yourself to run a lot of tests each month.

Blunder #6: The more variables the better

I hate testing too many variables at once, which is why I am not the biggest fan of multivariate tests. I’ve found when you combine all of the winning elements in a multivariate test, your conversion rates don’t go up as much as the software tells you they ought to.

If you modify too many variables at once, without testing them all, you also won’t know what variables are helping and which ones are hurting. For that reason, you should only try to make one change at a time.

A good example of this is Tim Sykes’ sales letter. Two weeks ago, he changed the video, the headline, the copy and even the design of the form fields. In the end, he had a huge drop in conversions.

That doesn’t mean the test was a failure. What it does mean is that Tim didn’t know which elements of the new design boosted conversions and which ones decreased conversions. To get a better understanding of this, he should have tested each variable independently instead of testing them all at once.

Blunder #7: Testing micro conversions

Do you remember the blunder at the beginning of this article in which Hubspot tested click-throughs? A click-through is an example of a micro conversion… Basically, Hubspot tried to increase the number of people moving from one part of the funnel to the next.

A macro conversion in that same scenario would be testing if a change impacted conversion rates. So instead of just testing the button color to see if it impacted click-through rates, the macro version of that test would have been testing if the button color impacted conversion rates.

Now, this doesn’t mean you shouldn’t look at micro conversions, but you should focus on macro conversions while keeping micro conversions in mind.

Blunder #8: Launching a new design

I know I said this post only contained seven blunders, but I had to throw in an eighth one…

The biggest blunder you can make is to redesign your website because your design is outdated. I’m a big believer that you should never just redesign things, but instead you should continually tweak and test your design until it’s to your customers’ liking.

At the end of the day, it doesn’t matter what you think of your design. All that matters is that it converts well. If you jump the gun and just change things up because you want to, you can drastically hurt your revenue.

Similar to the example in blunder Blunder #6, Tim Sykes also launched a new design because he wanted something fresh. Within hours of launching the design, he noticed that his revenue tanked. There was nothing wrong with the code; everything seemed to work; but the design wasn’t performing well. So he had no choice but to revert back to the old design.

You have a website to gain business from. Creating a new design won’t always help boost your revenue, so continually tweak and test elements instead of just redoing your whole design at once.

Conclusion

It’s ok if you make mistakes while you optimize your site for conversions. As long as you learn from your mistakes and avoid making the same ones over again, you’ll be fine.

I hope this blog post didn’t discourage you from running A/B tests on your website. Instead, I’m hoping that it encouraged you to run tests the right way.

Are there any other big mistakes you should avoid when optimizing your site for conversions?

If you want to break through to real profits online, you need some serious firepower. For a limited time I’m sharing some select tips and tricks Amazon, Microsoft, NBC & Hewlett Packard paid thousands of dollars per hour for, FREE.
  • The step by step guide to monster traffic generation
  • The how-to guide for increasing conversions on your website
  • 7 Cashflow killers your analytics tools are hiding from you
     
 
100% privacy, I will never spam you!

Comments

  1. I love articles on A/B testing. There’s things you can change (for free) that will boost conversions and profit. I think the biggest mistake is not letting a test run long enough. I don’t think you need to have 100,000+ views a month to run A/B tests but you need to be uniform in what your sample size is. So if you test once with a set of 5,000, run all your tests with a set of 5,000 views. Just be consistent with the data.

    -Amir

  2. Another mistake that a lot of businesses are doing is to treat traffic building and conversion rate optimization (or in this case, A/B testing) as separate. Some even assign micro-KPIs for the teams (eg. CPC for traffic team – don’t laugh, it’s happening out there) and this will definitely sabotage a campaign as a whole.

    Different ad copies, search terms or sources of traffic deliver visitors with a wide range of expectations. If the teams do not have a common goal, the expectations of visitors may not match well with what you are trying to send across with your landing page.

    In some cases, the silo-ed teams may even have ongoing optimization activities on both sides.

  3. One major problem with A/B testing is that there will never be conclusive results. Companies who go into testing thinking they’re in pursuit of the pot of gold have already lost,.

    Instead, go into testing KNOWING that the testing will never end. There’s just too many variables constantly at play to determine a page indefinitely perfect.

  4. Thanks for the great advice we just launched a new site wish we knew this before! But we’ll start tweaking it the way you mentioned instead of doing a completely different redesign.

  5. Awesome points, Neil! It’s really important to test for a decent period of time and not do too many tests at once.

    I think part of the reason #8 is so important is that people tend to get used to a design and regular visitors are the ones converting the most. When we sold our old site and the new owner redesigned it in the first 2 weeks, many of our regulars complained. Not a good way to make a first impression!

  6. Great run-down Neil, even throwing 8 tips for the price of 7 for good measure ;) It’s a timely post as we’ll be venturing into LinkedIn ads soon and these A/B testing tips will come in handy. Interesting point you made about how people re-designing their website should do so gradually if running A/B tests as I our web design agency places a lot of emphasis on user experience during the build process, but I guess having a firm understanding of what worked and didn’t work in your old site can only enhance things? Thanks for sharing!

  7. Neil, all good advices. In our environment many of them proved to work well. I like your focus on the conversion metrics rather than on the click thru rate. At the end this is what matters for business. Focusing on a single variable is priceless and not easy. Its always tempting to change this and that at the same time, but…. By changing two or more things at once you shoot yourself in a foot. You will never know what impacted the results. Two simultaneous changes may neutralize each other, which may result in no increase or decrease. By replacing a single word in a subject line in our email marketing campaigns we managed to increase overall open rate AND click through rate. It does take time to choose one version over another with confidence. I agree that it’s not a free game though as many like to think. Running a loosing version just as a test means revenue losses for business. That’s why it makes sense to change things gradually little by little. It’s also bad for SEO if a website is constantly undergoing some drastic changes.

  8. I don’t completely agree, here are my comments.

    Blunder #1: Believing what you read

    This has to do more with people thinking that there is some sort of magic green or red button that works in all cases. If you’ve done any testing at all you will know that there is no such thing.

    Oftentimes the right color is contextual. This is usually done by people who don’t have even a hypothesis and testing strategy to test and are just throwing crap on the wall to see what sticks.

    Of course that wouldn’t work.

    Blunder #2: Ending tests when they are statistically significant

    I see your point.

    But when it comes to the numbers, I prefer to have a good grasp of the fundamental statistical theories that are behind the tool, rather than have a rule of thumb.

    For most of the testing tools, they are using classical statistics and doing standard hypothesis testing. And they do this by approximating the binomial distribution by a normal distribution by means of the central limit theorem.

    The textbook way is to have n>30, which I have found to work fine in practice and provide sustained results when the tests go “live”.

    However, the numbers only tell part of the picture. There are a wide variety of validity threats that can affect the test and make it essentially rubbish for telling if a variation is better than another.

    Blunder #3: Expecting big wins from small changes

    Maybe. There is a difference between changing a small element that affects the thought sequence of a user, and changing a small element that does not do that.

    It’s pretty possible to get a big win from a small change.

    The real question is, WHY are we testing this small change? If there is no reason behind it, it’s probably going to do nothing.

    Blunder #4: The first step in A/B testing is to come up with a test

    Agree 100%.

    Blunder #5: Running a lot of tests on a regular basis

    Same thing here. Running a lot of tests is not the problem.

    Running a lot of tests with no strategy, no big picture of WHY you’re testing this or that; that is the problem.

    Blunder #6: The more variables the better

    Lack of strategy. Same thing.

    Blunder #7: Testing micro conversions

    If the micro conversion makes sense and helps to move the person forward in the funnel, then an increase in that is worth testing, IMHO.

    However, I would always keep an eye on the macro conversions (whenever possible).

    Blunder #8: Launching a new design

    I don’t agree. If the design looks like it comes from the stone age, a complete redesign will usually result in increased conversions.

    However, it’s the same thing. Slapping crap on the wall to see what sticks, or because it looks “beautiful” does not work well, redesign or not.

    There are ways to try to make this a bit less stressful. Usability testing, etc.

    I would split test the new design vs. the old one on an uneven or even split depending on comfort level, instead of just switching it over suddenly.

    Oh well…didn’t mean to go on for so long, but I’m really passionate about A/B testing.

    Typo in Blunder #2 BTW, “rule thumb”.

    • You make a good point of split testing an old design with a new design. I think it’s easier to split each page at a time or even specific elements on a page… but you could do a whole site versus another site.

      I’ve never really tried it out, but I’ll consider it for the future. My issue is, you can waste a ton of time doing that versus slowly adjusting, tweaking and testing your way into a new design.

  9. I’d say that even if the red button in the HubSpot report DIDN’T increase conversions, it doesn’t mean it wouldn’t in other situations.

    In my 10 years+ online, I realized, often the hard way, to take everything with a grain of salt. Especially from marketers.

    The only REAL piece of advice anyone can offer is that the proof of the pudding is in the eating.

    And I think telling people not to up and change their whole design is flawed advice too.. and would be a “blunder.” You can change the entire design and revert back just as easily as a button color.

    I changed the design one of my sites and conversions went through the roof. On another site it killed conversions, but in all of 5 minutes I was back to the old design.

    • 1. Yes the red button or any test could help or hurt… just depends on the site and situation. Results can vary a lot.

      2. I am not saying changing up your whole design can’t boost conversions. But it isn’t a systematic way of increasing conversions. Sure you could put back the old site with a click of a button (assuming you don’t have a ton of custom stuff), but you could be wasting a lot of time. If you spent that time continually tweaking, learning, and testing your current design I think it would be time better spent.

  10. Thanks, I will definitely do some of these.

  11. I’ve always been a big fan of “keeping it real” when it comes to CRO. Coming from a science background, I learned that short term results don’t are useful if they can’t be duplicated.

    There are outside factors that can skew numbers as well. The strangest things can cause surprisingly misleading data. While doing CRO for a Hotels, we found that sporting events could cause the home/visiting team’s colors to perform much better/worse than normal for a given week. Only to shift back in following weeks.

    Also, looking at a customer lifetime value can also make a huge difference which was semi-touched on with the Hubspot reference. 21% more click through (or even conversion) doesn’t mean 21% more sales – and 21% more sales doesn’t mean 21% more profit. Margins, both for one transaction and customer also should be factored to make the best long term decision.

  12. I need to find good plugin for A/B testing… never did it before

  13. frankly speaking i heard first time about A/B testing and still trying to understand what does that mean.

  14. I also wanted to comment on launching a new design…

    There is nothing wrong with wanting a fresh design. Many sites do need a modern face lift. Site owners just need to know that conversion rate optimization MUST be done after a new design.

    We published a case study that showed how we increased revenue for a client by 300%. He had big plans for growth, but he was on a hosted platform that limited the things he could do with the site. We suggested he move his site to a better platform and we would develop a cleaner design with better features. This was after we got the 300% increase.

    After we launched the new site, sales tanked. Almost all of the progress we made with the old site was lost. We jumped into CRO mode. We analyzed traffic to see what was going on. It took some time, but now he is having record sales again. Changing to the new platform is allowing him to grow and do unique things.

    No one wants an out dated site. It is bad for branding. When people come to a site, what will they think if it looks outdated?

    Typically, the drop in conversions after massive design changes are due to repeat visitors getting lost because things are different and it is due to overlooking design flaws that effect conversions. These can be fixed. You dont want to be stuck with an outdated design because there is growing pains involved. Staying modern pays off in the long run for your overall brand.

    Joe

    • Joe, great points. Interactive design and switching things up often is the key to catching eyes and visitors. It is essential to not get caught up with complacency and the same design for a long period of time. Thanks for reading and your feed back :)

    • Great points and something that I didn’t consider. I’ve just completely redesigned my site to make it look professional and match the brand ideals and I’ve noticed a few big changes in site useage statistics. I couldn’t quite work out why but it might have something to do with new users getting lost in the traffic.

      I think it’s time to start getting feedback and seeing where things are going wrong.

      And great article Neil. Conversion rate optimisation is my next big step so it’s great to read some of the basics to make sure I don’t start making the same mistakes.

  15. Good stuff. One other point about point number 1. I’ve read on your blog that it was the colour that increased the conversions in the red / green test. But I’ve read on other blogs that it was the fact that this colour was different to the reset of the site (i.e. there is quite a bit of green on the site in the logo etc) that drove conversions not the fact that it was red.

  16. Thanks Neil, I’ll surely avoid those 8 blunders while testing and optimizing my site. Thanks a lot.

  17. I’m not so sure about Blunder #7 – Testing Micro Conversions, Neil.

    Let’s use a real-life analogy. Assume that Kissmetrics’ marketing team is supposed to generate leads and the sales persons are supposed to close them.

    If marketing generates loads of new leads (their micro conversions) but sales’ closing rate goes down because they are overwhelmed, then the problem isn’t with either. It’s just a problem of not having enough salespersons.

    Another scenario could be marketing generating new leads but closing rates dropping because they’re not exactly great leads.

    —Long-winded analogy ends now—

    Each element in the funnel has a job to do. If an element is doing its job well but things screw up elsewhere, don’t punish the element. A red CTA button might increase clickthroughs but overall macro conversion of free-trial signups remains the same, that doesn’t mean you punish the button, it means you ought to look elsewhere… probably the traffic mix that is coming in.

    • Yes, but you can measure if the leads are qualified for the reps. So we’ve ran into this issue before and as long as marketing is driving high quality leads, sooner or later sales will catchup as they add more people to their team.

      You do make a valid point, but you typically can look at macro conversions from a department basis.

  18. Certainly testing couple of weeks is going to make The Decision.

    Best regards from I. C. Daniel

  19. Happens everywhere, some how, true.

  20. Hey @Neil What Happen to your blog, it was showing error yesterday?

    By the way good article.

  21. I totally agree about the “don’t believe what you read” point. Especially if you then, for instance, go Tweeting/liking/or writing an article about something if it is complete nonsense! Always do some research; in this day and age it’s easy and only takes minutes. Look for proper, acknowledged sources. I find Wikipedia gets a bad name of it and can actually be an easy way to find proper sources.

  22. Hi Neil!
    A/B test is completely new for me. Although, this post is helpful to understand. Thank you for your sharing.

  23. never heard of a/b testing. going to try it in my blog and thanks for sharing it.

  24. Hey Neil,

    I just found your work about two weeks ago. I have learned quite a bit in just two weeks. However, I am not sure I understand what I should be looking for with A/A testing. Is it as simple as running the same version of your website to ensure each test produces similar results?

    Thanks for all of the wonderful information.
    Rick

  25. Whoa to #8! I just did a total overhaul to a Volusion site. The old design was like 6 years old. The new site looks better, but sales tanked. Wow, interesting to read that is a common blunder. I’m really not sure what to do now.

    Also, I just started experimenting w/ split tests in emails. I usually send 2 articles per email and have always noticed that the first article gets more clicks. Often way more clicks. I did a test where I reversed the order of the two articles, and it did confirm that people seem to click on the first article regardless of which one it is. Maybe I need more data because I have nowhere near 100k people on my list.

    Also, part of the reason I did the big overhaul is that Volusion is really tough to deal with. Do you have any other suggestions for an ecommerce platform for a site that has a few hundred products that is easier to work with on the design side?

    Thanks for the great post.
    Nick

  26. Experiences thought you things which you can not learn just by reading any crap of piece available on net ofcourse reading articles on Seomoz, Quicksprout, Search Engine Watch and Mashable helps but some people take things so seriously and are too afraid to take actions even i was one of those but i think we need to experience as well to get the perfect ideas of what works and what don’t

  27. I agree with Alvia. Experience is key. Experience cannot be learned through a book or even this blog.

  28. You make some excellent points in this article, Neil.

    Two questions:
    1. How do you know how long is long enough?
    2. What would you advise small companies that don’t yet have a lot of page views every day/week?

  29. well, Most of the blunders which experts make is that they stop making changes when they thinks that some ROI has started coming. They should make changes and see improvements in each and every design.
    Thanks

  30. One major problem with A/B testing is that there will never be conclusive results. Companies who go into testing thinking they’re in pursuit of the pot of gold have already lost,.

  31. How statistically significant do you think the data should be? I’ve heard that its more important to know how accurate your data is and how to interpret it over trying to reach that golden 95% statistically significant number.

  32. Most of what the writer here is saying is very surface and very basic stuff. Especially on his last point, totally incorrect and terrible advice. Of course you should redesign your site periodically. You can’t AB test your page into the best possible page. Sometimes, industry themes, design standards completely change and you can’t just roll design changes to an existing page.

    People — don’t forget, you can’t AB test your way to the best business model or perfect design. Sometimes you have to take meaningfull risks. You think Nokia could have AB tested their phone into an iPhone??? What the *real* experts say is to take leaps of faith in what you believe and come up with several new designs and models, and do continuous improvement on those over the year. But you should absolutely put a stake in the ground and do redesigns at least once to twice a year. Don’t believe the hype people.

  33. Good article; I enjoyed reading it.

    Not sure I agree with the sentiment about tweaking and refining towards a finished design. I definitely agree with that after a design has been reached, but sometimes radical departures must be made, as a narrowing path of refinement may miss great ideas slightly off of the path of refinement. Optimizely are big proponents of exploratory behavior prior to refinement.

    Cheers for the article.

    • Simon, thanks for the feedback. You bring up some great points. We as marketers and designers must always iterate for maximum exposure and conversions. Thanks for reading and I look forward to hearing more from you :)

Speak Your Mind

*