What Spending $252,000 On Conversion Rate Optimization Taught Me

ab testing

Over the last year and a half, I spent a total of $252,000 on conversion rate optimization. That money was used to hire three firms: Conversion Rate Experts, Digital Telepathy and Conversion XL.

All three firms produced a positive return on investment, but if I knew what I know now, I would have made a lot more money. Although conversion rate optimization (CRO) in its simplest form is finding out why your website visitors aren’t converting into customers… and then fixing it, there is a lot more to it.

Here is what I learned by spending $252,000:

Lesson #1: Gather qualitative and quantitative data before you start testing

You probably already have several ideas on how you can boost your conversion rate, but before you start testing them out, there are a few things you should note:

  1. Do not run tests based on your gut. You’ll end up running a ton of tests that fail. Instead, make sure you base all your testing on data.
  2. Numbers don’t paint the whole picture. For this reason, you need qualitative data… so don’t be afraid to ask your current and potential customers questions.
  3. It’s better to have more data than less. For this reason, you should spend a month gathering and analyzing data before you start testing.

When I first started paying CRO companies, we just tested based on our Google Analytics data. It wasted our time and money as it didn’t produce an ROI. Since I started gathering both quantitative and qualitative data and started making decisions based on it, the ROI has been huge. The $252,000 in spend has turned into millions in additional revenue.

Lessons #2: Do A/A tests before you do A/B tests

When I first started A/B testing, there were instances in which the new variation had big increases in conversion… upwards of 30%-40%. However, when I analyzed revenue increase, I didn’t see much, if any, increase at all.

Do you know why?

The software I was using for A/B tests was not great. This is why you should run an A/A test before you run an A/B test. In the A/A  test, you take the original variation of your landing page and test it against itself. After a few hundred conversions, if the conversion rates aren’t very similar, it means your software is probably off.

It’s really important that you run an A/A test first as this will help ensure that you don’t waste time with inaccurate software.

Lesson #3: Don’t expect increases on a monthly basis

When I got into A/B testing, I expected increases in conversions on a regular basis, at least every other month.

Boy, was I wrong!

From my experience, you tend to get only a few wins each year that drastically affect your revenue. Those wins typically make up for all of the fees you pay to consultants. This is really important for you to understand if you have cash flow issues as you will be out a lot of money before you make it back. CRO is a long-term investment, not a short-term one.

In other words, don’t expect to make a return on your investment within the first three months. Expect to start seeing a return in about six months. By the end of twelve months, you should be cash flow positive on your CRO investment.

Lesson #4: Multivariate tests never work… or at least for me

If you don’t know what a multivariate test is, check out this article.

In a multivariate test, a web page is treated as a combination of elements (including headlines, images, buttons and text) that affect the conversion rate. Essentially, you decompose a web page into distinct units and create variations of those units. For example, if your page is composed of a headline, an image and accompanying text, then you would create variations for each of them. To illustrate the example, let’s assume you make the following variations:

  • Headline: headline 1 and headline 2
  • Text: text 1 and text 2
  • Image: image 1 and image 2

Now that you know what it is, I recommend that you stay away from it. Every time I’ve taken the winning elements from each multivariate test and made them the default version, I’ve never seen the increase in conversion that the testing tool is showing I should get. I’ve tried this with multiple tools and have had statistically significant results, but the results never equated into huge revenue increases.

You can try multivariate tests, but I personally never find them to work in my favor.

Lesson #5: Don’t optimize for conversions, optimize for revenue

Most CRO consultants focus on increasing your conversion rate, but they don’t focus on increasing your revenue, which, at the end of the day, is all that matters.

When you are running tests, you’ll quickly get an understanding of how a conversion decrease can increase revenue. The quickest way to do this is to increase your prices.

For example, assume that you sell flowers online. Out of every 1,000 people that visit your website, 5% convert into paid customers. Because you charge $10.00 for each flower you sell, you end up making $500 in revenue for every thousand visitors.

Let’s assume you decide to increase your prices to $20.00 per flower. Due to your increased prices, now out of every 1,000 people that visit your website, only 3% convert into paid customers. In this scenario, you make $600 in revenue for every thousand people that visit your website. Even though you conversion rate went down from 5% to 3%, you still were able to make more money by increasing your prices.

So, when you are working with CRO consultants, have them focus on optimizing your revenue, not conversion rates.

Lesson #6: Focus on macro conversions, not micro conversions

The difference between macro and micro conversions is that macro focuses on the big picture while micro focuses on the small picture.

An example of a macro conversion would be how many people end up buying your product. A micro conversion would be optimizing how many people click the “add to cart” button or view your “pricing page”. As you already know, just because someone added something to his or her cart or viewed your pricing page doesn’t mean he or she will purchase your product.

When running A/B tests, don’t run tests that will boost your micro conversions as that will not guarantee a boost in your macro conversions. Focus on macro conversions such as increasing the total number of sales, instead of optimizing how many people view your pricing page.

Lesson #7: Drastic changes = drastic results

Once you optimize your conversions by making all of the major changes, you’ll notice that small tweaks – from headlines to button colors – stop having huge impacts on your conversion rate.

It’s not that those small tweaks aren’t important. It’s just you’ve taken care of all the low hanging fruit that is stopping people from converting. At this point your best chance of boosting your conversion rates or, more importantly, revenue is to make drastic changes.

From changing up your signup process to forcing people to sign up for a free account before you upsell them, you have to make drastic changes.

Many of these changes won’t work out, but some will have a positive impact. Just get creative as that is the trick to boosting revenue. For example, a drastic change I made that tripled my contact requests was changing my contact page to an infographic.

Lesson #8: Don’t forget to optimize your back end for conversions

When you think of optimizing conversions, what comes to mind? The concept of turning more visitors into customers, right?

Although that is CRO, it doesn’t mean it has to stop there. What about increasing the lifetime value of your customers? An example of it would be getting them to spend more money with you or getting them to refer their friends to you.

There are a lot of things you can do to boost your back-end conversions, so don’t just focus on the front end. In many cases, it is easier to optimize your back end than front end, so focus on both.

CRO consultants, from my experience, love working on the front end of your website, but they can also do wonders for your back end… so make them work on both.

Lesson #9: Consultants aren’t miracle workers, they need direction

That’s right, CRO consultants aren’t miracle workers. Just because you are paying someone six figures a year to help you boost your conversions doesn’t mean they will actually produce results.

If you want to get the most out of your CRO consultant, here are a few things I recommend doing:

  • Require that you have a call at least once every two weeks.
  • Assuming you have enough traffic, make it a requirement that you have to run at least two tests a month. It’s a numbers game after all.
  • Make them gather and analyze quantitative and qualitative data every quarter. What your customers have to say changes over time.
  • Have them focus their efforts on creating wireframes and writing copy. Most CRO consultants are slow at design, so might as well have them focus their time on what they are best at.
  • Don’t expect your consultant to come up with all of the ideas. You know your business better than anyone else, so make sure you throw your ideas out there. Our biggest conversion increases came from ideas my business partner and I had.
  • Make sure you are working with a consultant who is good at executing. What I’ve experienced is that tons of consultants understand the concepts of CRO, but many of them suck at execution. If they suck at execution, nothing will get done.
  • Once you find a consultant you really like working with, prepay them for future work and ask for a discount. You should optimize your spend. We do it with all of the consultants we work with, and it easily saves us over 20% a year.

Lesson #10: Just because you had huge wins doesn’t mean you will see large revenue increases

Even if you are using good A/B testing software, focusing on optimizing revenue and measuring macro conversions instead of micro, it doesn’t mean things will go the way you want.

Over the last 1.5 years, I’ve noticed a trend that just because a test says it increases your revenue by 30%, it doesn’t mean it will maintain that increase in the long run. I am not 100% sure why, and nor are the consultants I work with as they have seen this happen too.

Even with statistically significant tests, those 30% revenue lifts tend to be 15% lifts in the long run. My best guess is that there are other variables that come into play such as the change in quality and/or volume of your traffic over time.

This doesn’t mean you should discount those tests or stop testing. Instead, this means that you need to test constantly and work on optimizing your conversions/revenue. It’s a never ending game.


Have you started optimizing your conversions yet? If not, I hope this post encourages you to start as CRO can help you make millions.

Before you can start, you’ll need to have two things: traffic volume and conversions. If you have fewer than 10,000 monthly visitors or $200,000 in yearly income, it may be hard to optimize your conversions.

On the flip side, if you have over $500,000 in yearly revenue, then you should consider making CRO a line item expense. Just like you would pay a bookkeeper or accountant each year, you should constantly pay a CRO consultant.

What have you learned by optimizing your website for conversions?


  1. $252K??? I could have bought a house….

  2. Amir @ Blue Mile Media :

    This is a great article and I don’t think people work on their conversions as much as they should. My experience is when I had been making money with Adsense and wanted higher CTR.

    I always focused on trying to increase traffic but sometimes it’s easier to increase conversions then to increase traffic. Testing different things like color, placement, wording, etc is fun and sometimes it’s surprising what actually has the best conversions that’s why you should never stop testing things out.


    • Thanks Amir,

      There are definitely some cool things you can try when testing. Don’t be afraid to get creative and take risks. Most importantly I agree that you should never stop testing.

  3. A brilliant article Neil, as usual!

    And as Amir says above, it is very surprising to see what the little changes such as colour change, placement, wording can do to your overall conversion.

    Once again, great post!

  4. Scott Bradley :

    Awesome article Neil.

    I love how you break down what you learned down into bite-sized pieces.

    The biggest takeaway I got from this article is to always focus on the moving the needle for the end result you want to impact the most (sales/revenue). Focusing on the micro stuff is a waste of time (button clicks, pageviews of a particular page).

    Thanks for providing such a valuable post Neil. This is going in my reference folder in my Evernote!

    • I am glad you got so much out of this post! Thanks for letting me know what specifically you found helpful.

  5. I have done some conversion rate optimization in the past but I have never worked with a consultant on it before. The list of things to do are going to be useful. Thanks, Neil.

    By the way, I also tried multivariate testing in the past, on a smaller scale of course, and I find that it doesn’t work for me too (in the biz opps niche). With limited time and resources, simple A/B testing is the way to go for me.

    • Great, give these tips a try and see how they work out for you.

      Yep, I don’t think it will work for most people.

  6. So out of the three companies you have used who would you recommend? Also you mention the software you used was not very good, what would you consider a very good software?

  7. great article neil…
    i am a regular reader of peep s blog and he has some very clever articles about conversions… i have also read crazy egg case study on CRE blog.. very impressive article… !
    most people spend their majority of time on seo and getting more traffic… but few like you care and spend time and money on things that are extreamly essential for any online or offline business….

  8. Thanks Neil! This article is really timely for me as I’m in the process of hiring one of these consultants right now. Care to name any of the split-testing software that you found consistently unreliable?

    • I haven’t found any that are “consistently” unreliable but I have found Google experiments has worked really well.

  9. Lesson #5 – 5th paragraph, 2nd line should start with ‘Due’.

    Good insights, as always, Neil!

  10. Man this is inspiring! Often we lose lot of time & resources in little details of multivariate tests. Focusing on drastic changes for a change would be a refreshing way to look at CRO.

  11. Very insightful and useful post, even for somebody who’s just starting out in the “making money online” biz. Thanks Neil! 🙂

  12. I need to read this again but I really appreciated the A/A test.

    Just imagine the rubbish a multivariate test is producing if an A/A test comes out bad!



  13. Great post Neil. The combination of lessons #1 and #7 really resonate and I think are complementary. If it was just about making changes based on gut feelings we would have already done it by now. The real insights and the big wins seem to consistently come from talking to customers and prospects and challenging our own gut feelings. We try to use quantitative data to determine the areas to focus on, and then the qualitative data to uncover the key insights.

    Thanks for sharing.

  14. I don’t want to be a pessimist here but I can’t resist pointing out to the fact that there are more than 2 typos in this article.

    Excellent info though 🙂

  15. I see so many case studies showing huge results in things like click through, leads and sales, but they never show the most important number, revenue. That is the most important.

    Crazy Egg will be posting an article on a case study we did which was a very unorthodox. We broke several of the typical steps in CRO and went old school. We just focused on the sales/traffic and revenue/traffic as we made changes. The increase in revenue and profit was significant.

    We only used data from Analytics and NO A/B software. We just simplified the site and used common sense to improve it. The lesson is; you don’t always need to overload yourself in data unless your site is already finely tuned. The more tuned in your site is and the better the conversions, the more data you need to improve it further.

    Neil’s sites are tuned in very well so he will need as much data as possible to improve things further.

    Neil, I love the idea of A/A testing. Google Content Experiments is driving me nuts with their multi-armed bandit testing. It would be interesting to A/A test with it. Never thought of that.

  16. Great Job Neil as always, If I could add anything it might be a #0 – folks have to get past their own conceptions and ego. CRO works if you are willing to use it correctly – along the lines of #1 and 2 above. On #8 we recommend using a human testing to help back up quantitative results. I have also seen a fade in results as mentioned in #10. I chalk it up to competitors – that we can never count on sitting still while you are increasing 30%! 🙂

  17. Scottsdale Computing :

    This is a great article Neil, like always. I think lesson #3 is very important. When analyzing your conversion rates and data, even when doing A/B testing, don’t expect anything to happen overnight. It is a granular process. Like you said, within the first 6 months is when you should hopefully start seeing positive results.

  18. Aditya Todawal :

    Awesome post Neil. Sometimes even client done know what do they want from their website. This is a great help. I liked the ideas of A/A test before A/B, Optimize for revenue instead of conversion and lifetime value of customers. I learn a lot from your blog. Thanks!!! 🙂

  19. What the Hell 😮 when i saw the headline i fainted for 5 seconds literally :O really 😮 $252,000 , man….if i had that much money i would have left blogging :O and sat on himalayas and became a sanyasi lol 🙂 still man it was a good read 🙂 enjoyed as always 🙂

  20. I’m loving your articles on actual testing you’ve done and the results. Of course we can’t just transfer your results without doing our own testing, but it helps to seed ideas.

  21. Great post Neil, thanks for sharing the inside scoop on your findings.

    I agree that split testing yields better findings when you have a minimum (10K+) solid monthly stream of traffic to put against it. Even if a site doesn’t have 10K visitors, I think split testing is still worthy but tests should be much simpler, run longer, and conclusions shouldn’t be made until a lot of clicks and visitors have happened. Too many people conclude too early, let the tests run.

    Its also interesting to split test based on the source of traffic, organic, referring, ppc, etc. They sometimes react differently but it takes a lot of volume to start testing different sources but if you have the stream, why not try.

    • Hope my findings help out with your own testing. I like your point on split testing by traffic source, thanks for bringing it up.

  22. I’d love to know what CRO tools you say ‘don’t work properly’ – that information alone is valuable if it’s correct in every case.

    And the one snippet of advice everyone should take from this is ‘optimise for revenue, not conversions’.

    That’s great advice and worth remembering. Cheers!

    • I haven’t found any that are consistently unreliable but I have found Google experiments has worked really well.

    • I also like Google Experiments, but I hate how they push more traffic to a variation if it starts to outperform. This can give inaccurate results.

      I think there must be a bug in it. Many people have been complaining about this.

      Don’t let Google Experiments push too much traffic to one variation, especially if the conversion rate is not much different. If you have a test where variationA has 1000 visitors and variationB has 200 visitors, but the conversions are not much lower (20-30%), then restart the experiment. It is not fair to variation B because it has a much smaller sample size so the conversion rate will not be as accurate. 200 visitors is nothing and Google Experiments should not be deciding it does not deserve a shot after such little traffic.

      I wish they had a 50/50 option. For now, let the tests run for long period of times and don’t let one variation eat up the majority of the traffic. If it does, just restart the experiment. When all testing is done, add up all of the traffic and conversions from all of the copied tests to get more accurate numbers.

      • What they are doing is smart. It actually makes the testing more accurate. This helps them figure out how stable the conversion rate is, which affects the overall outcome.

        • Yes and no. The problem is Google Experiments is making the determination too soon. Here is an example from this past week…

          We are testing 3 variations of a landing page. After 3 days the Original has 1291 visitors with a 21.69% conversion rate. Variation 1 has 140 visitors with a 12.14% Conversion rate. Variation 2 has 358 visitors with a 20.11% conversion rate. Variation 1 only had a .22% probability of outperforming so Google is giving that page no traffic. On just the third day of testing, Variation 1 had a only 5 visitors and a total of 140 visitors after 3 days. How can anyone know how that page converts on such little traffic? Sure, the original is being testing to see if it is stable, but what do we know about Variation 1? Nothing. We had to restart the test to get more data on Variation 1.

          On the second attempt, after just 4 days, the Original received 1436 visitors with a 20.96% conversion rate. Variation 1 received 16577 with a 21.30% conversion rate. Variation 2 received 331 visitors with a 17.82% conversion rate. Now, Variation 1 is showing that it actually has something, but Google would have never let us know that with 5 visitors per day going to the test due to their original assessment that Variation 1 only had a .22% chance of beating the original. Wow, talk about a bolt of lightning according to Google. The fact is, Google never gave Variation 1 a shot in the first test due to their multi-arm bandit testing. It determines loosers too quickly.

          The test is currently running and now it seems to be stable with reasonable traffic going to each page. We are going to let it run another week to get solid numbers. So far, if we add up the numbers from ALL of the tests, the Original and Variation 1 has the same conversion rate. I will probably discount Variation 2 soon and just let the other 2 run. I actually think Variation 1 will win by the end of next week.

          By the way, all of the pages are pages we designed including the Original. We are just trying to improve it further.

          This issue typically happens with high converting pages such as free signups, but we do have issues sometimes with normal converting sites. We just ran a test that took over 2 months because the Original was taking the vast majority of the traffic in the first month which left us with insufficient numbers for the Variation. Finally, after a month of testing, the Variation began to catch up and get more traffic to see its real potential. Now it is outperforming the Original after 2 months of testing.

          I am just saying, EACH test page needs significant traffic to see what is going on. No page should be discounted after a small sample size.

          • Thanks for sharing the example… you make some really good points.

            I guess I am going to start looking for another testing solution to start using.

            If you have any other ideas on good testing solutions, let me know.

            • Sorry for the late reply. You need a subscribe to comments option here.

              I usually use Google because it is easy and everyone is already set up with it. I just need to restart tests if they get out of control.

              The 3 other testing tools I would try are Optimizely, Visual Website Optimizer and Unbounce. Just make sure when you set up tests with them that you don’t use their internal editor for creating new pages. Create pages locally, upload them, then set the actual page url as your b version.

              I like having a completely separate url for testing so I can better track differences between pages in Google.

              • No problem, thanks for the suggestions.

                • Just saw this thread… if anyone like Neil needs a good comparison of low cost website testing tools, I put together a comparison article here (and you will see why not to use Google Content Experiments!) Hope it helps…

  23. I liked the info graphic idea to boost More contact requests. These golden nuggets is why I read your posts

  24. Great article, I always learn something new with you, this time I learnt that ROI for CRO comes in the long term, I thought that in a period of 2 or 3 months you could get all benefits.

  25. Daniel Gonzalez :

    This is a killer post Neil. Interesting insights on what you prefer CRO’s to focus on.

    It’s also interesting to see the disparity between people’s expectations when working with a CRO, and your views after spending a year and a half doing it.

    Sweet man!

  26. Neil,

    Qualitative data is a great point that is often overlooked. You might find half the respondents choose a product because of cost savings whereas the other half choose because of time efficiency. That’s a great starting point to test headlines stressing two different benefits to see what appeals to the masses. This approach is way better than testing some magic word that you read about that improved one company’s conversion rate 67%!

  27. Honestly, the content is too good on this site.

    I’m leaving…just kidding

    I spend most of my day talking about hubspot, seo, inbound marketing, and neil.

    ————–my girlfriend hates it—————-

  28. I especially like your input about optimizing for revenue rather than merely the CR. I think most consultants will have a hard time with that one, since they’re often all about tools and setting things up correctly and less about the business-side of things – at least in the form you describe it here. That’s an important input.

  29. Marvin Charles :

    Key takeaways:

    1. Do A/A testing before A/B testing
    2. No multivariate testing

    BTW, Conversion XL has a great blog!

    Thank Neil

  30. THANKS, Neil, for another TERRIFIC post!

    As someone with an advanced degree in Statistics, I’d like to make a point of clarification re item # 4 about Multivariate tests. What is described (e.g., headline 1 vs 2, text 1 vs 2, image 1 vs 2, etc) would be more appropriate to call “an experiment with more than one treatment effect.” Of course, it does not have to be just headline 1 vs 2. It could be headline 1 vs 2 vs 3. It still would be one “treatment effect,” but with more than two “levels.” As for “multivariate,” that in statistics refers to a totally different thing. In experiments, using designs with more than one treatment effect is not typically done, for the reasons Neil described. Although there are ways around, at both the design and at the data analysis stages. But the term “multivariate” is used for different things.

    “Multivariate” in this context would be having in the model the treatment effect of interest (or more than one treatment effect, but that’s unusual) AND other variables. These other variables could be other “fixed effects” (e.g., day of the week, gender, etc), could be covariates (continuous variables), and could be what’s called “random effects.”

    PLEASE, it’s not just semantics! 🙂 People go to lawyers, accountants, doctors, etc, but not when it comes to statistics. I used to work in a statistics consulting office, so literally hundreds of clients later, this for me is one of those “don’t get me started!” subjects 🙂

    • Thanks Olga for the clarification. I will look into it further but I am sure I can trust your description of it.

  31. Neil, brilliant post. You really know how to go inside in your audience mind. How the heck did you do that?

    Please advise. 🙂


  32. Great article Neil.

    What gets measured gets done, so be careful what you measure!

  33. Matt Ackerson :

    I love this article, Neil.

    Lession #10 stands out in particular as it reminds me of why many CTR’s for banner and text ads decrease over time (e.g. the medium / format becomes banal and no longer unique for viewers, so clicks drop). This is one reason why Google is continuously changing it’s search ad formats and presentation.

    Keeping the presentation and format fresh over time (in this case, for a conversion funnel design and user experience) seems to be a formula for long term success with sustainable results.

    Again, great article, thanks for writing it.

  34. Funny, I’ve been researching conversion rate optimization intensely for the past week and Neil just so happens to write this post… right when I need it most.

    That the second time you’ve done this and its never just fluff to induce an opt-in.. it always extremely insightful and actionable.

    Thank You, Neil. Im studying marketing and hungry for more in-depth blog posts (or even a pdf) on this particle subject.. please.

    • Glad this post has reach you at the right time. I don’t but you should check out conversion-rate-experts.com

      • Yes, great recommendation. I’ve already read their website from front to back, examined their methodology and downloaded all their e-books.

        I stumbled upon your explainer video tutorial last week where you mentioned them. I’m reading conversionxl.com as this very moment. Peep’s blog is filled to the brim with value and keen insights in conversion optimization.

        I spent 3 weeks researching conversion optimization and their blogs never popped up once. I probably wouldn’t have them if it weren’t for the equally valuable blog here.

        Again, thank you, Neil.

  35. David Gadarian :

    As usual drop dead obvious and very sharp. A/A testing. Love how you are always questioning everything.

  36. It’s interesting that multivariate tests never work. I always just assumed that this was THE way to test.

  37. Hi Neil,

    I truly hope that your blog readers get the immense value you have provided in this article, well written and factual content.

    Over the last 3 years we have completed in excess of $4.5Mil worth of conversion optimization work for our clients and in isolation this seems like a staggering amount but as you so correctly stated it is not what you spend but what you get back in return as a multiplier on your conversion optimization investment.

    Most of our clients start small so they are risking less and then based on the measurable return they received they then do more and more until it is just part of their business lifecycle.

    Please listen to Neil when he tells you to do an A/A test as this has been a value step for us in determining how accurate the A/B or Multivariate testing tools are at one point we used a certain piece of software that showed us a 36.7% improvement @ 92% certainty the only problem was it was the same page on both sides of the test… 2 years later then provider has still not answered us on the reason why.

    Hence we no longer use them.

    Thanks again Neil awesome article!

  38. Hi, Neil.

    You just inspired me to write a post about your Lesson #10, “Just because you had huge wins, doesn’t mean you will see large revenue increases”.

    I experienced this very same phenomenon as Intuit Global’s Web optimization leader… and I wrote about what we learned here:


    I’d love to hear your thoughts on what I’ve shared.

    Thanks for another awesome post!

  39. I never done AB testing but still getting good results 🙂 Neil my logic is to complete your homework first then put the things go live.

  40. Are adwords conversions high enough to break even costs?

  41. Toni Aničić :

    This is by far the best article I saw this year about CRO.

    I’m more into e-commerce specific CRO, but this article has some nice takeaways for me and my clients as well.

    • Mihai @ Freshome :

      Neil – one suggestion. For the comments section you should install the subscribe to comments plugin …so people can get an email each time someone replies to their comment. People come back to follow the discussion, and you build the community.

    • Thanks Toni, glad you could still takeaway some tips from this post.

  42. Ashok Khandekar :

    Brilliant article Neil,

    But your last 2 paras makes this article for (may be) 1% of all the IMers !!! What about the remaining 99% ? Do you have any guidelines / solution for them ?

    Please guide us too Neil.

    Thanks & cheers !

    • Sadly these are the guidelines I have, I know they may not work for all marketers but hopefully more people will start getting into A/B testing.

  43. Reasearch is always the most expensive part but you usually get a decent ROI to paid back your research and test!

  44. Jhon @ Career Guidance :

    Hi Neil,

    Thanks for sharing with us your experience about conversion rate! from my little experience, the best results I have got was when I made small changes, when I worked on details. when made major changes, for example at design and functionalities the result wasn’t that one I expected. I think this tips is really powerful: “Don’t optimize for conversions, optimize for revenue”


  45. Another great post Neil! It was a big amount of money spent! But i guess your learning from that money was really worth it. thanks for sharing with us!

  46. One reason we often see a difference between test results and ongoing results is the “newness effect”. You test something new, your returning visitors notice and clickity-click, your stats go up, you roll out the winner… and now your users get used to it and their response or conversion rates decline. You can mitigate this by waiting for the numbers to stabilize before interpreting the results.

  47. Phew! $252K thats a big sum and will take a while for me at least to have that in my pocket.

    But yes, useful insights into CRO, something refreshing. Like Samay mentioned above that too much focus is on SEO and traffic, but such type of material is nearly absent.

    Neil Pateil is nothing short of a genius..uh..oh..he already knows that 😉


  48. Sorry. I cannot afford $252k to learn these 10 things. You saved my money.

  49. Josh Patterson :

    Interesting idea about A/A testing. Will have to give this a go.

  50. HI Neil! Excellent tips and I love reading the post and I got one question to ask from you. What you mean by back end conversion?

    Thanks for sharing such a worthy experience 🙂 though.

  51. Hi Neil. We recently started using A/B testing on our website and couldn’t believe how CRO could make such a difference to our sales- too many people concentrate on getting website into the traffic but not actually converting it. Thanks for the excellent article.

  52. Chris Neumann :

    Hi Neil,
    I do CRO for several SaaS companies similar to yours, and in my experience, if the site doesn’t have a lot of low hanging fruit on the home page, then the biggest gains in revenue (thank you for making that a key lesson!) is in what you call “backend conversions”. The reason is that generally there is no one in a company focused on it. Usually the marketing person is responsible for getting people to the site, and the product manager is busy building features for core engaged users. There is generally no one helping onboard and convert signups or trials to paid users. As a result, there is generally a lot that can be done there.
    The other thing I’ve found is that a lot of gains can be made in simple headline testing, so anyone doing this on a DIY basis can generally test headlines pretty easily. The key is to have a headline which goes to the core problem you’re solving. Think “1000 songs in your pocket”. At the time the iPod came out, that was amazing – the alternative was carrying around a CD player, which obviously didn’t fit in your pocket.

    • Thanks for your feedback, I appreciate it. Always enjoy having a professionals added input.

    • ” then the biggest gains in revenue (thank you for making that a key lesson!) is in what you call “backend conversions”

      Excellent post Chris, very insightful. Any recommendations or perhaps a frame work you would use to help fix a businesses backend. Maybe a physical therapist or carpenter.

  53. Never heard anyone say not to optimize for conversions.. but rather for revenue. Interesting..

    • Try it, revenue is what you are ultimately striving to increase and increasing conversions doesn’t always produce a higher revenue.

  54. Great article Neil, you make some really good points here. I think you may be being a little bit brutal with your dismissal of multi-variate testing however as we undertake this quite often and see good results from it. Granted, A/B testing often does seem to create better results however I do believe there is value to multi-variate testing. Definitely agree with focusing on revenue too, there’s no point getting great conversion rates if it isn’t going to make the client any more money!

  55. Thank you Neil for article every-time i visit here i find something new this time it was CRO.

  56. Dennis van der Heijden :

    Hi Niel,

    I just want to share part of my presentation on the Conversion Conference this week, that matches you learning. In Convert.com’s A/B testing tool we found the following:
    – CRO agencies (that only do CRO) you can see 1/3 tests successful
    – Then online agencies that part of their business do CRO 1/5
    – The ‘rest’ 1/7

    On top of that you see that a very small lift is the most common lift (1-10%).

    The difference in CRO agencies is usually the preparation time to get to a test. Did you find a link between the research (pre-work) of the agency was correlated with the lift and success?

    RPV (revenue per visitor) is great number to test on. Many clients keep focusing on conversion rate, but RPV is a key metric that more and more agencies are taking at the one metric to determine the success.

    I saw the case of CrazyEgg on the CRE site and was impressed lots of good ideas there.

    Looking forward to hear what the differences were in agencies approach although thats difficult on a post where you call them out all 3. But did you see a correlation between pre-test research and test success?


    • I did find a high correlation, as you saw on the case study they tend to do a lot of prep-work before they do their test.

  57. Really great article that touches basically all important points that usually fall under the table (e.g. A/A-Test or that only drastic changes will lead to drastic results). This article alone has earned you a new follower 🙂

  58. Thanks for sharing Neil. I feel at this point of my business, split testing is the most important thing I can do marketing-wise.

    These guidelines will help.

  59. Great Post!!

    “Lesson #4: Multivariate tests never work…” << yeah, that's a painful lesson from my past experiences : )))

    Really great and informative post… Thank You..

    I disagree on "Lesson #5: Don’t optimize for conversions, optimize for revenue" though.

    I mean, your point is correct, but it strongly differs on a business niche and many other factors, so to rely on this #5 lesson, without considering the numbers game and factors related to your specific business type – would be a mistake.

    to quote your example:
    "assume that you sell flowers online. Out of every 1000 people that visit your website, 5% convert into a paid customer. And because you charge $10 for each flower you sell, you end up making $500 in revenue for every thousand visitors."

    this could work for online flower store business ( could ) but definitely wont work for consumer electronics, as niche is extremely competitive and every cents count when customers compare the prices online.

    Once again, yes in general I agree with you, I just wanted to say that #5 is not a general rule, and business owners have to consider many factors, before they decide to follow this rule : ))


  60. Online Mastering :

    jeez man! thats a lot of money! …pretty baller, lol

  61. I loved your point about optomizing for revenue not conversion. I this this very simple key is missed by a-lot of poeple who want to see the easy numbers instead of taking time to see the real numbers. I also agree with your statment optomize your backend for conversion…you have to get the fish in the boat before you have really cought it.

  62. What do the CRO consultants, “that execute”, charge?

  63. Great article!

    Thinking aboht the 30% revenue increase being more like 15% in the long term – is this isolating new customers from existing ones? If not, it’s probably a mix issue as conversion increases are less for existing users…

  64. Wordpress Installation Service :

    I think you make a good point about conversion rate optimization only being at the level of at least 10000 uniques or $500000 p/a revenues. Trying to pitch such a service to smaller companies just doesn’t work! I’ve tried! 🙂

  65. Thanks for this article. It helps me to get a good idea about conversion rate optimization (CRO)

  66. Great post Neil,

    I would like to ask you about CRO consultants.

    Are they handling all the design while testing or they simply suggest things to change and your designers have to do the job?

  67. @252,000 to increase conversion rates? Gosh, that’s some serious money! Anyway, thanks for the great tips! They would be helpful for me in the long run.

  68. Ramsay from Blog Tyrant :

    Hey Neil.

    Wonderful post.

    So what are your thoughts on Optimizely?


  69. Arlington Hardwood Floor Care :


    You have given great ideas on CRO (conversion rate optimization).
    Thanks for the article Neil.

    Arlington Hardwood Floor Care

  70. Hi Neil,

    Just wondering about whether point 6 should apply to all scenarios. The specific case I’m thinking of is a mobile site where people may be less inclined to purchase (a long buying process or more expensive/complex product). One assumption is that they come back later and buy on desktop/in-store. Do you think it makes sense to test for micro conversions in that situation?

  71. RE #5 Optimizing for revenue in relation to raising prices. It’s important to consider the annual/lifetime revenue value of a customer, as it applies to your business. In your example if the annual revenue was $25/year, then each customer would have an additional $15 left in that year. Take the incremental $15 x 20 additional customers and that’s $300 + $500 (the initial $50 x $10) = $800. Or 30 customers at $20 = $600 + 20 x an incremental $5 (for the $25 annual total) = $150 for $750 total. And that’s in year 1, the lifetime value is likely to be much greater.

    Don’t get me wrong, I think the point to optimize for revenue is a great one, but make sure you consider the value of having a customer as it applies to your business. Depends on what you are selling, how often, and for how much. But once you have a customer, the data on what they buy, their email, a relationship etc. you have the opportunity to continue to market to them which is less expensive than acquiring a new customer.

  72. Good post!

    I work as a mathematician/statistician for a company building a product search engine for ecommerce.

    There are a few problems with tools such as google experiments and many others, which may explain some of the problems you’ve had with results not continueing to perform after the test, and that multivariate tests have not worked.

    Google Experiments is pretty bad at calculating statistical significance, which is surprising since it’s.. well.. google.

    When you look at a conversion rate, and try to find a significance level, you need to make some assumptions about the structure of the converion rates. Such assumptions may for example be “all conversions are independent of each other”, “all visitors are equally likely to convert”. These are examples of assumptions made if you model conversions as being drawn from a binomial distribution, which seems to be what google is doing.

    There is an obvious error in the last assumption “all visitors are equally likely to convert”. On the web, people have a ton of different goals, and most of your visitors may never intend to convert (according to your conversion goal). If this assumption is breached, the variance of the test may skyrocket, and you need to find other ways to model the conversion rate that are more realistic. An easy way of adding the possibility for visitors to have different conversions rate is to view the conversions as being drawn from a binomial distribution where the conversion rate (the parameter in the binomial distr) is modeled as coming from a probability distribution itself. Check out beta-binomial distr for more info.

    The second problem is that in statistics, point estimates are almost always worthless. “The big button converts better than the small button”, but what is the interval? Is it that the big button converts 0-50% better? So if you change it’s not that unlikely that the change is close to 0%? What if the assumotions in the modelling of the conversion rate is wrong (which it always is), and the interval is actually -20%-70%? The big button may actually be worse.

    A multivariate test is nothing special, it’s basically just a bunch of simultaneous A/B tests. Often the same assumptions are made, that the conversions are binomially distributed. So you have to take the same thing into consideration, that the variance will probably be much higher than many tools will report, and will take much longer to converge.

    A multivariate test requires much more data, of course. If you have 2 button sizes and 3 button colors you have 6 variations of the page and all visitors will be divided into one of the 6 instead of one of the 2 as in a A/B test, requireing much more time for the results to be significant.

    So 2 things:

    1) Always assume that the variance is MUCH larger than the tools report.
    2) Never look at point estimates of significance. Look at the interval instead.


  73. Definitely an informative post. If I were to take one learning away from your post, I would say #7 would be the most important.

  74. I’ll buy make a condo with that money.

  75. BP from Flipnet :

    Amazing post. When being talked about CRO, I trust only articles that emphasize importance of business background for the CRO expert. It has little to do with the color of your button, when you are able to have in mind the whole picture. Thank you!
    And thanks to Mike. Great comment.

  76. You are a liar and an asshole!

    • If you ask any of the consultants I linked to above, they’ll tell you that I paid them… they know the results as well.

  77. Hi!,,,,,,,,,,,

    Hello!,,,I liked the info graphic idea to boost More contact requests. These golden nuggets is why I read your posts..Thank you for your post!,,,,,,,,,,,,,,

  78. Sanjay Shenoy :

    Hi Neil,

    Really nice post! I personally believe that #5 is a really good point. I think a lot of people get carried away by increase in conversion rates and then realize it is not affecting their revenue at all.

    One other way of looking at this would be to decrease the costs instead of increasing the revenue I guess. Because not every customer would be happy to digest the fact that there is an increase in price. Instead of putting the costs on the customers wouldn’t be better to save and pass on the savings to the customers? Again this might depend on which industry you are working on.

    Also, I guess once your sales increase ( as a result of conversion optimization ) your cost per unit will decrease which will therefore increase your profit. ( again this not be feasible in every industry )

    I am just thinking out loud here please feel free to correct if I am wrong.


    • Yes, you are correct and that would work. The big issue with it is that it is tough to save on costs, but if you can do it that is ideal as you can pass some of that savings onto the customer.

  79. Hello !
    This post has encouraged me too much. Now I am excited to optimize my conversions. Obviously this post would be a guideline for me. I really liked that “do not expect increase on monthly basis”. This is actually true.

  80. What are some tips that can increase your macro conversion? (Refferncing to the example in your article) If I decided to focus on buying the product rather than the add to cart button. Wouldn’t I do things like add better images, product description, etc. In turn that should increase the add to cart conversion rate and product sales in the process?

    • Here are some articles I’ve wrote in the past that will help:






  81. Hi Neil,

    Heads up there is a grammatical error in your article on Tech Crunch:

    “In addition to uncovering instances of duplicate content that could be hampering TechCrunch’s inability to rank well in the natural search results.” ‘Inability’ should just be ‘ability.’

    I hate it when there are errors on my pages that go unnoticed for long. Just wanted to help. You my dude, love your site.

  82. Hi Neil,

    I cannot understand the last paragraph…

    On the flip side, if you have over $500,000 in yearly revenue, then you should consider making CRO a line item expense. Just like you would pay a bookkeeper or accountant each year, you should constantly pay a CRO consultant.

    Smaller sites cannot do conversion optimisation, split tests and all that good things we read in your fabulous blog? Or you do you meani to hire high-end consulting firms $$$,$$$?

  83. Neil, great article. Perhaps not strictly related to the article, could you advise which factor has more weight in terms of sale conversion rate: streamlined checkout (i.e. less questions, perhaps PayPal) vs. discounts/points/rewards at checkout?

    • It really depends per business. In most cases streamlining won’t convert as well as discounts or rewards. But you will have to a/b test it.

  84. Always you explain about CRO comparing the real time aspects so that i like to follow your thoughts ever.

    After a long time i am gonna to revamp my client site according your points of getting ROI with practically through the CRO. Thanks Neil 🙂

  85. Thanks Neil – my biggest takeaway for immediate action is “don’t optimize for conversion, optimize for revenue”. This just cuts the chase for me – after all, the clicks and conversions are all preliminaries to the final end and most times we get lost chasing the in-betweens that we forget why we are supposed to be optimizing. Again, thanks.

  86. seo raleigh nc :

    each time i used to read smaller posts that as well clear their motive, and that is
    also happening with this post which I am reading at this time.

  87. my singing monsters cheats :

    Good day very cool blog!! Guy .. Excellent .. Wonderful ..
    I’ll bookmark your web site and take the feeds also?

    I’m glad to seek out numerous useful info right here within
    the submit, we’d like work out extra strategies on this regard, thanks for sharing.
    . . . . .

  88. Timely article, thanks for sharing 🙂

  89. Azizul Yusof :

    Lesson #5: Don’t optimize for conversions, optimize for revenue

    How about both? Just let the people that handling CRO concentrate on the CRO and the business owner concentrate on the Revenue.

    There are many ways to increase the revenue but if you are looking on the Price increase and the Conversion decrease because of that price increase, then the people handling the CRO needs to find a way to maintain or increase the Conversion ie. better targeting.

    Because the situation has changed (Price increase), CRO effort shouldn’t stay as it was before, it also need to change.

  90. Hi Neil,

    Great Post on CRO! But I’m little confused… What is the best conversion rate for a site such as review site? Also, what is the optimum bounce rate? Could you please tell.

  91. Bryant Jaquez :

    This is something consultants should print our and give to their clients before they start a relationship 🙂

    Neil, thanks for this one. I agree with every point.

  92. Nice post. Most of the people talks about conversion rate of ads. This post really helps them.

  93. Hai Neil
    What Conversion Rate Should Be for a Web Design Company and Some One please tell me how to increase my website Conversion Rate

  94. Clash Of Clans Triche Gemmes Illimite Gratuites :

    Juste être sûr que vous êtes sur le bon site web, vérifiez l’adresse en déambulant le nom This Means
    War Triche de l’écran ainsi que de profiter des gratifications
    offerts k?rester ce merveilleux outil!

  95. Best makita drill :

    Hi just wanted to gife you a brief heads up aand let you know a few of the images aren’t loading correctly.

    I’m not sure why but I think its a linking issue. I’ve tried it in twoo different
    web browsers and both show the same outcome.

  96. Thanks for every other great post. The place else could anybody get that kind of info in such a perfect way of writing?

    I’ve a presentation subsequent week, and I am on the search for such info.

  97. It is not for many who need to arrange several servings
    of coffee all in the same time. Along with one-touch technologies,
    you’ll have precise control over your personal brewing (and there’s no grinding or messy clearing up afterwards).
    Also, regular ground coffee should stay in contact with all the water for between six or eight min.

  98. Florence Coultrap :

    Thanks for your post. Another issue is that just being a photographer includes not only difficulty
    in recording award-winning photographs and also hardships in acquiring
    the best digicam suited to your needs and most especially situations in maintaining the standard of your camera.
    This can be very true and obvious for those photographers that are in capturing the particular nature’s interesting scenes :
    the mountains, the actual forests, the actual wild or perhaps the seas.

    Going to these exciting places absolutely requires a
    digicam that can surpass the wild’s tough areas.

  99. I know I hate doing conversion and even tracking. Don’t even mention sifting through the data to find something useful.

    Absolutely detest it.

    Having said that, I know when I was working with PPC a lot back in AdWords’ big days that small changes could mean big differences. And when I say “small changes”, I mean changing the headline, which may be actually macro change rather than micro as the article mentioned.

    How often I saw a failing ad/campaign jump to become a massive producer with a simple change. But the thing that drives me up the wall is the unforeseen or invisible nature of the potential changes. You just never know when it’s good enough or if there is room for improvement.

    You make some changes and keep optimizing. Then when you think it’s done, you think of something else you can try instead – something in a completely different direction. When you start thinking this way, the process never ends and you end up with hundreds of projects all needing to be tracked tested again and again and leaves no time for anything else.

    And if all that produced greater profit, that’s a whole other story. But the sad truth is that most of the time, the testing results in a decrease in income. It’s only when you find a winner from a test, you can capitalize on it by scaling it, at least in the case of PPC campaigns. Of course, then you want to test it with some more tweaks or something in a totally different direction.

    Oh, how much I hate conversion…

    Sorry for the rant. This article just brought back so many memories….

  100. We arrived too late in the day to drive out on the salt flats.
    Gowdy has been an active participant on a number of committees; among them the Committee
    on Oversight and Government Reform, the Committee on Education and the Workforce and the Committee on Judiciary.
    At our next club meeting Kris approached me about joining with him to race one of his Harley Sportsters at Bonneville.

  101. Jacqualine Chiquito :

    I have created several online businesses first and created the plan second for additional guidance and direction. I do believe that a plan is essential in the long haul and you should have one at some point. As the saying goes, “If you fail to plan, you plan to fail”.

  102. Hey Neil, Great Article. Just got here from your CRO Post on Forbes.

    You said Not to expect results on a monthly basis.
    My question is how long should you run each test before you conclude its failed.

    Would you keep tests between 2-4 weeks for a website that’s getting traffic as much as quicksprout or would you hold tests for 6 months before you stop?

    Here’s the link to Neil’s recent Forbes article for others.

    • Assuming you have enough conversions, thousand a month… if the test isn’t close to statistical significance… under 80% in 4 weeks, I usually I end the test and call it a failure.

  103. Hey Neil what would you do if you traffic is high enough to do CRO but the number of conversion is low. For example when doing our own test, we had a hard time hitting statistical significance since we were only converting at 1% (about 60-70 orders a day).

    We still want to work with CRE, but having experiments with inconclusive data is what scares me.

    • At 60 to 70 a day it is enough to do testing. You can’t do a lot of variations though… just an A/B test. Go for drastic changes.

  104. Andrew Wise :

    Hey Neil, Thank you for another informational post.

    I totally agree with you with your lesson #3: Don’t expect increases on a monthly basis. Sometimes we tend to focus on the short term when we should focus more on the long-term goal.

  105. Thinking of using Conversion Rate Experts — what was your cost for them specifically?

    Pricing structure?

    Have a call with them tomorrow but impatient.


  106. I agree with you Neil on most points and would add a few pointers.

    From experience having worked with both entrepreneurs and big companies: Entrepreneurs usually have no brands (they are just starting out, or a niche) so their number 1 issue has always been traffic. My advice to them has always been to sell their stuff on the big platforms with the best conversion rates e.g. Amazon, eBay (for auto parts etc), and then lure away the customers with discounts and other services to their own shopify/woocommerce sites.

    Big companies (big brands) do not usually have a traffic problem but a huge conversion problem. To compound the problem even further, to implement CRO fully and benefit massively, you have to have the top senior executives fully behind CRO i.e. willing to test and make drastic changes if there is data and expert opinion backing it up – big enterprises are woeful at making any changes. They love reports and more reports but making changes is slow and chaotic at best.

    If an entrepreneur or startup company I suggest you look for a marketer focused on growth and not just incremental growth but 30% type growth that Neil refers to (Macro-conversions, the big hitters). Usually they are called growth hackers, but whatever you choose to call them – they should know how to acquire customers (ppc) but most importantly convert those customers.

    If you have high conversions rates, you can double the amount you spend on acquiring customers!

    Let that sink in for a moment.

    You can spend more on PPC (CPC/CPA) if you have impressive conversion rates – why? Simply put you are confident that whoever clicks on your adverts will end up being a customer. So doubling your PPC spend means (having a high quality score because of impressive conversions) being able to outbid the competition each time you bid for keywords/advertising space.

    Most people think its all about getting traffic but if your website is a leaking bucket in the long run it will cost your business.

    • Great points, all. Thanks so much for sharing your experience and knowledge with everyone else. That’s what makes this community great!

  107. Even if you are using good A/B testing software, focusing on optimizing revenue and measuring macro conversions instead of micro, it doesn’t mean things will go the way you want.
    Nice Post Thanks For Sharing!


Speak Your Mind