What Spending $252,000 On Conversion Rate Optimization Taught Me

ab testing

Over the last year and a half, I spent a total of $252,000 on conversion rate optimization. That money was used to hire three firms: Conversion Rate Experts, Digital Telepathy and Conversion XL.

All three firms produced a positive return on investment, but if I knew what I know now, I would have made a lot more money. Although conversion rate optimization (CRO) in its simplest form is finding out why your website visitors aren’t converting into customers… and then fixing it, there is a lot more to it.

Here is what I learned by spending $252,000:

Lesson #1: Gather qualitative and quantitative data before you start testing

You probably already have several ideas on how you can boost your conversion rate, but before you start testing them out, there are a few things you should note:

  1. Do not run tests based on your gut. You’ll end up running a ton of tests that fail. Instead, make sure you base all your testing on data.
  2. Numbers don’t paint the whole picture. For this reason, you need qualitative data… so don’t be afraid to ask your current and potential customers questions.
  3. It’s better to have more data than less. For this reason, you should spend a month gathering and analyzing data before you start testing.

When I first started paying CRO companies, we just tested based on our Google Analytics data. It wasted our time and money as it didn’t produce an ROI. Since I started gathering both quantitative and qualitative data and started making decisions based on it, the ROI has been huge. The $252,000 in spend has turned into millions in additional revenue.

Lessons #2: Do A/A tests before you do A/B tests

When I first started A/B testing, there were instances in which the new variation had big increases in conversion… upwards of 30%-40%. However, when I analyzed revenue increase, I didn’t see much, if any, increase at all.

Do you know why?

The software I was using for A/B tests was not great. This is why you should run an A/A test before you run an A/B test. In the A/A  test, you take the original variation of your landing page and test it against itself. After a few hundred conversions, if the conversion rates aren’t very similar, it means your software is probably off.

It’s really important that you run an A/A test first as this will help ensure that you don’t waste time with inaccurate software.

Lesson #3: Don’t expect increases on a monthly basis

When I got into A/B testing, I expected increases in conversions on a regular basis, at least every other month.

Boy, was I wrong!

From my experience, you tend to get only a few wins each year that drastically affect your revenue. Those wins typically make up for all of the fees you pay to consultants. This is really important for you to understand if you have cash flow issues as you will be out a lot of money before you make it back. CRO is a long-term investment, not a short-term one.

In other words, don’t expect to make a return on your investment within the first three months. Expect to start seeing a return in about six months. By the end of twelve months, you should be cash flow positive on your CRO investment.

Lesson #4: Multivariate tests never work… or at least for me

If you don’t know what a multivariate test is, check out this article.

In a multivariate test, a web page is treated as a combination of elements (including headlines, images, buttons and text) that affect the conversion rate. Essentially, you decompose a web page into distinct units and create variations of those units. For example, if your page is composed of a headline, an image and accompanying text, then you would create variations for each of them. To illustrate the example, let’s assume you make the following variations:

  • Headline: headline 1 and headline 2
  • Text: text 1 and text 2
  • Image: image 1 and image 2

Now that you know what it is, I recommend that you stay away from it. Every time I’ve taken the winning elements from each multivariate test and made them the default version, I’ve never seen the increase in conversion that the testing tool is showing I should get. I’ve tried this with multiple tools and have had statistically significant results, but the results never equated into huge revenue increases.

You can try multivariate tests, but I personally never find them to work in my favor.

Lesson #5: Don’t optimize for conversions, optimize for revenue

Most CRO consultants focus on increasing your conversion rate, but they don’t focus on increasing your revenue, which, at the end of the day, is all that matters.

When you are running tests, you’ll quickly get an understanding of how a conversion decrease can increase revenue. The quickest way to do this is to increase your prices.

For example, assume that you sell flowers online. Out of every 1,000 people that visit your website, 5% convert into paid customers. Because you charge $10.00 for each flower you sell, you end up making $500 in revenue for every thousand visitors.

Let’s assume you decide to increase your prices to $20.00 per flower. Due to your increased prices, now out of every 1,000 people that visit your website, only 3% convert into paid customers. In this scenario, you make $600 in revenue for every thousand people that visit your website. Even though you conversion rate went down from 5% to 3%, you still were able to make more money by increasing your prices.

So, when you are working with CRO consultants, have them focus on optimizing your revenue, not conversion rates.

Lesson #6: Focus on macro conversions, not micro conversions

The difference between macro and micro conversions is that macro focuses on the big picture while micro focuses on the small picture.

An example of a macro conversion would be how many people end up buying your product. A micro conversion would be optimizing how many people click the “add to cart” button or view your “pricing page”. As you already know, just because someone added something to his or her cart or viewed your pricing page doesn’t mean he or she will purchase your product.

When running A/B tests, don’t run tests that will boost your micro conversions as that will not guarantee a boost in your macro conversions. Focus on macro conversions such as increasing the total number of sales, instead of optimizing how many people view your pricing page.

Lesson #7: Drastic changes = drastic results

Once you optimize your conversions by making all of the major changes, you’ll notice that small tweaks – from headlines to button colors – stop having huge impacts on your conversion rate.

It’s not that those small tweaks aren’t important. It’s just you’ve taken care of all the low hanging fruit that is stopping people from converting. At this point your best chance of boosting your conversion rates or, more importantly, revenue is to make drastic changes.

From changing up your signup process to forcing people to sign up for a free account before you upsell them, you have to make drastic changes.

Many of these changes won’t work out, but some will have a positive impact. Just get creative as that is the trick to boosting revenue. For example, a drastic change I made that tripled my contact requests was changing my contact page to an infographic.

Lesson #8: Don’t forget to optimize your back end for conversions

When you think of optimizing conversions, what comes to mind? The concept of turning more visitors into customers, right?

Although that is CRO, it doesn’t mean it has to stop there. What about increasing the lifetime value of your customers? An example of it would be getting them to spend more money with you or getting them to refer their friends to you.

There are a lot of things you can do to boost your back-end conversions, so don’t just focus on the front end. In many cases, it is easier to optimize your back end than front end, so focus on both.

CRO consultants, from my experience, love working on the front end of your website, but they can also do wonders for your back end… so make them work on both.

Lesson #9: Consultants aren’t miracle workers, they need direction

That’s right, CRO consultants aren’t miracle workers. Just because you are paying someone six figures a year to help you boost your conversions doesn’t mean they will actually produce results.

If you want to get the most out of your CRO consultant, here are a few things I recommend doing:

  • Require that you have a call at least once every two weeks.
  • Assuming you have enough traffic, make it a requirement that you have to run at least two tests a month. It’s a numbers game after all.
  • Make them gather and analyze quantitative and qualitative data every quarter. What your customers have to say changes over time.
  • Have them focus their efforts on creating wireframes and writing copy. Most CRO consultants are slow at design, so might as well have them focus their time on what they are best at.
  • Don’t expect your consultant to come up with all of the ideas. You know your business better than anyone else, so make sure you throw your ideas out there. Our biggest conversion increases came from ideas my business partner and I had.
  • Make sure you are working with a consultant who is good at executing. What I’ve experienced is that tons of consultants understand the concepts of CRO, but many of them suck at execution. If they suck at execution, nothing will get done.
  • Once you find a consultant you really like working with, prepay them for future work and ask for a discount. You should optimize your spend. We do it with all of the consultants we work with, and it easily saves us over 20% a year.

Lesson #10: Just because you had huge wins doesn’t mean you will see large revenue increases

Even if you are using good A/B testing software, focusing on optimizing revenue and measuring macro conversions instead of micro, it doesn’t mean things will go the way you want.

Over the last 1.5 years, I’ve noticed a trend that just because a test says it increases your revenue by 30%, it doesn’t mean it will maintain that increase in the long run. I am not 100% sure why, and nor are the consultants I work with as they have seen this happen too.

Even with statistically significant tests, those 30% revenue lifts tend to be 15% lifts in the long run. My best guess is that there are other variables that come into play such as the change in quality and/or volume of your traffic over time.

This doesn’t mean you should discount those tests or stop testing. Instead, this means that you need to test constantly and work on optimizing your conversions/revenue. It’s a never ending game.


Have you started optimizing your conversions yet? If not, I hope this post encourages you to start as CRO can help you make millions.

Before you can start, you’ll need to have two things: traffic volume and conversions. If you have fewer than 10,000 monthly visitors or $200,000 in yearly income, it may be hard to optimize your conversions.

On the flip side, if you have over $500,000 in yearly revenue, then you should consider making CRO a line item expense. Just like you would pay a bookkeeper or accountant each year, you should constantly pay a CRO consultant.

What have you learned by optimizing your website for conversions?