Neil Patel

I hope you enjoy reading this blog post. If you want my team to just do your marketing for you, click here.

7 Lessons I’ve Learned from CRO Tests that Generate Hundreds of Millions of Dollars

Conversion rate optimization is like the banana peel in those really cheesy video clips.

It’s pretty obvious it’s there.

We all see it.

But we all know it will trip you up when you get too close to it.

Because that’s the punchline, right?

Except tripping up on optimizing conversions for your business isn’t such a funny joke.

Not only is failure keeping you from growth, but it could also be pushing your business towards the red.

And while you might acknowledge conversion rate optimization (or CRO), it’s definitely not something you should be afraid of.

Instead, you should chase it just as much as you do anything else in digital marketing, if not more.

That’s why I want to share with you some of the hard lessons I’ve learned from my own failures with CRO.

I want to save you the silly slips that end up on your gag real.

And shortcut your way to the real money-making deals.

If it ain’t broke, do I need to fix it?

I sometimes have the question posed to me, in some form, whether I truly think conversion rate optimization is such an essential task.

In short, yes.

To elaborate, I believe that your business is built around conversions.

The more often a visitor to your website’s business completes a goal, like clicking through a landing page, the greater your chances are of making a sale.

And when you have someone convert by purchasing a product, you just made x amount of money because of a conversion.

Every click is an increase in the potential for your business to grow, so optimizing clicks is vital.

So calculating your current conversion rates is the first step every entrepreneur should take when discussing CRO, and thankfully it’s a pretty basic principle.

In this example, we have three page visits but only two conversions, which means our conversion rate is at a strong 66%.

It won’t likely stay there as page visits grow, but you see the basics of how we calculate conversion rates.

Actions divided by sessions equals conversion rate.

Optimization kicks in when you take the same page you’ve been tracking and start changing elements to see if you can get more conversions.

This illustration helps show the process.

Variation A has a healthy 23% conversion rate, while variation B dips to 11%.

This tells you that you’re better off going with variation A for this audience.

Your next step would be to find a new way to run this test with the same element, or change to a different area of the page and see what you can find.

The more you optimize and combine your winning elements, the better your conversion rate will become over time.

And the potential benefits of CRO, when done properly, are pretty astounding.

if CRO is done property

I don’t know about you, but a 49% increase in conversions would be a very impressive increase for my business.

And taking the average revenue per visitor up from $3 to $4.50 might not sound like much, but it’s a jaw-dropping increase when you start crunching numbers.

But let’s get to the meat of this issue.

What can you do to streamline your conversion rate optimization and start improving revenue?

I’ve worked on CRO projects that have generated hundreds of millions of dollars.

And to help you learn from my successes and failures, I’ve boiled down the basics to help you understand what to look for and why you should look for it.

Here are the lessons I’ve learned the hard way. so you don’t have to.

Lesson #1: Play for the long-term

Let’s get this out of the way now.

All of this advice can help you generate millions of dollars in revenue for your business.

But these results will not happen overnight.

Or maybe even this year.

The reason for this is that conversion rate optimization is all about long-term gains.

When I first started focusing on building my CRO strategies, I initially thought I would see conversion improvements all the time.

I was wrong.

While I did see small improvements from time to time, the drastic jumps I had hoped for came infrequently.

Even when I spent money on consultants, I would question whether my investment was worth the small gains I made.

But then I looked back through my years of data and realized that there were periods of drastic improvement.

And when I ran the numbers, I saw that my revenue more than made up for the price of my consultants.

That brought me to a realization.

This is all about long-term persuasion.

CRO is not a get rich quick scheme.

It’s a be patient, work hard, and get rich scheme.

It’s all a game of establishing lifetime value.

Lesson #2: Gut-testing will be gut-checking

I’m all for business owners following their gut when it makes sense.

Just not when you’re running tests.

I have good reasons for this too.

Mainly, you’ll likely just waste your time and money with unnecessary failures that could have been avoided by a little pre-test data gathering.

But perhaps more egregiously, you’ll be ignoring the insight of your existing and potential customers.

Alienating your customers is the last thing you want to do.

The big takeaway here is that more data is better.

randomly chosen elements ab test

If your ideas take less than five hours to test, there’s a good chance that you’re not devoting enough time to collecting the data that will give your testing direction.

Which means you may need to spend weeks on gathering information about relevant ideas before you make a change.

Remember how I said this was a long-term process?

So gather as much information as you can beforehand to give direction to testing.

One way you can do that is with a simple customer survey.

would you lie to see xyz

Using open-ended, multiple-choice questions like this in a survey of your existing customers can help pinpoint elements that contributed to their conversions.

You can then use that information to change the elements that your existing customers claim would have helped them.

For example, if you get significant feedback that your landing page’s headline wasn’t a big factor in conversion, you may not need to test it much.

But your gut reaction may have initially been to start with your headline.

See what I mean?

Don’t just jump in based on your gut. Gather data and move with precision.

And if you do want to start making changes immediately, I recommend using proven conversion optimization methods to optimize your conversions before you leap into testing.

Try conversion scent methods for example, like with this retargeting ad.

As you can see, this method maintains the “scent” of your digital marketing by creating and maintaining the same look and feel across all of your efforts.

You can use this stylistic approach to help conversions and improve the consistency of your optimization testing results without making gut decisions.

Then, when your data gathering is complete, you’ll be able to test new methods.

And you can even test the effect of a method like conversion scents as you branch out, too.

Because it may not work best for you.

The point is, you don’t know until you have enough data to tell the difference.

But for the beginning, my advice is to rely on best practice and gather data to inform your testing.

You’ll see more improvement faster by leaning on the successes of others and building your own methods on top of that.

Lesson #3: Performance is important

One of the elements I see slip a lot when businesses perform CRO tests is their overall site performance.

What they don’t know is that this can inadvertently skew their data and make their test results ultimately useless.

Why do I say that?

Take an element like load times, which are incredibly important for increasing conversion rates.

When you let a load time bump above four seconds, you’re already creating a huge spike in your bounce rate.

Or, an equally huge dip in your conversion rate.

And it’s a fairly common occurrence amongst marketers to inadvertently increase load times when adding imagery, copy, or any other new element to a page.

When Moz performed an extensive redesign of one of their landing pages, they noticed a significant increase in load time as well.

To help mitigate the effect they used a few tricks to make it easier to fully load their page even on a slow connection.

The results were pretty amazing.

They were able to shrink their load time by 40% and provide a 17% lift in the overall performance of their landing page.

All of this was based on one performance oriented element.

Pretty wild, right?

But load time isn’t the only thing you should consider.

I also see a lot of businesses ditch underperforming keywords and landing pages because they don’t see a reason to put effort into an inherently low performer.

That’s a million dollar mistake to my mind.

I think your underperformers are just showing red flags that call for a little extra attention.

Take AutopilotHQ for example.

They took their underperforming keywords and decided to revamp their approach completely.

They created individual landing pages and performed A/B tests for each.

When they spent some significant time with each, they saw drastic improvements.

The kicker here?

It only took a week, and they saw huge increases in conversions.

So don’t just abandon your poor performers.

Give them more attention, and make sure none of the technical elements of your website are working against you.

Lesson #4: Test your A/B testing software

There’s an assumption out there that all A/B tests are created equal, but this isn’t so

When I first started A/B testing, I had a few strange instances where I saw drastic increases in conversion but no additional revenue.

This wasn’t because my sales process was flawed, it was because I had inaccurate A/B test results.

How could that be?

To be honest, I’m not sure why any A/B tester would purposely try to skew a user’s results.

You rely on accurate data to make informed decisions, so getting unreliable data from a test will just persuade you to use a different A/B testing platform.

Either way, I always recommend testing the initial variation of your landing page as both elements of your A/B test to tell you if your A/B tester is telling you the truth.

I call this the A/A test.

If your results look even remotely different, like this graph:

You should probably switch testing software immediately.

Here’s how you could do this on a platform like Optimizely.

When you open up your dashboard on their site, you’ll want to start a new test.

Click on the Experiments tab, then create a new A/B test.

Now, you’ll need to fill in some basic information about your test and include the URL of the landing page you’ll be conducting your test on.

Once you input this information, you’ll be back on a dashboard that looks like this:

As you can see from my notes, you’ll want to leave both your original and variation #1 alone.

The idea is to test identical pages for their similarity, thus testing the accuracy of your actual testing software.

To get started, you’ll have to add a Metric.

You can choose one metric or many.

For now, let’s just test a metric related to general page visits.

You’ll be given the option to track an increase or decrease in total conversions, unique conversions, or a few other options.

Once you’ve added a metric to your experiment and saved it to the project, you’ll be able to fire up the actual testing.

start experiment in optimizely

Now, you’ll be able to track the performance of your single page using the two “variations” we’ve created.

Since they’re the same page, you should get very similar results from both.

They might not be exact, but as long as they’re close, you know you have an accurate testing tool.

And there are tons of A/B testing tools out there to choose from.

Use one that’s both helpful and accurate.

You’ll save time and money.

And you’ll be making informed decisions with the degree of certainty that only testing can give.

Lesson #5: Test one variable, and only one

When scientists run tests, they look for something called “causal relationships” in their data.

In simple terms, that means they look for how the piece of data related to or caused another in their experiment.

To control the flow of these relationships, they will only change one variable at a time while keeping a “control” variable the same.

But when it comes to digital marketing, I see the very common mistake of testing multiple variables all the time.

It’s even promoted as a winning feature by many testing platforms.

I’m not so sure they’re right.

This practice is known as multivariate testing, or the idea that testing multiple elements of a landing page, email, or anything else can help you increase your conversion rates.

My biggest grief with this is that it encourages businesses to test too many elements at once, thus nullifying any tangible results from the test.

Here’s an example from that shows how a multivariate test works.

Let’s say this is the original email.

You have a very clear understanding of your email, its performance, and the tendencies of your audience.

But on email B, you change both the time of day you send the email, the formatting of the email itself, and the subject line.

That’s three new elements.

Not surprisingly, the results are different.

It just so happens in this case that your open rate and click-through rate see a decline. Bummer.

But now what do you do?

In this case, they went with the original email for their final list of receivers.

And what a win, they got a better response across the board.

My issue here is that the final email is the exact same as the original email, but now you have a higher open and click-through rate.

What did you learn from email B?


Because you don’t know what caused the potential change.

For all you know, one of the two lists is a group of outliers, and you can’t trust your data to make a decision.

I see this problem every time I examine data from tests that contain multiple variable changes at the same time.

The data I got back was ultimately useless.

And the further you dig into the logic of this, the more my point makes sense.

If I change two elements and see a 15% increase, did element A cause the increase, or element B?

Was it a combination of both?

Instead of getting answers, your questions compound exponentially as time goes on with no actionable results.

Change one headline. Test.

Change one body of text. Test.

Use this data to your advantage, and don’t confuse yourself.

Lesson #6: Remember your ultimate goal is revenue

A lot of talk goes toward conversion rate optimization without acknowledging the fact that we’re really focused on revenue optimization.

You want more money because that keeps you in business and lets you achieve your goals.

And even though it might negatively affect conversions, sometimes revenue optimization takes you on a different path.

Like raising or lowering prices.

Let’s say you sell 100 products at $50 at a 5% conversion rate. That would make you $5000.

But if you put that same product on sale for $45 and sell an extra 50 products, you just made $6,750.

But you didn’t change a single element of your landing page or email. You just changed the price.

What I’m trying to convey (again) is that your overall focus needs to be on your long-term goal.

Incremental wins in conversion rates will eventually lead to big wins in revenue, like when I helped Adore Me boost their revenue by $5.6 million.

quicksprout case study

This means focusing on the big picture, or macro conversions over micro conversions.

A macro conversion is a big picture goal like buying a product.

Micro conversions are small wins like clicks and email address that don’t necessarily equal revenue.

If a test boosts micro conversions but not macro, find out why.

If it’s a permanent effect, drop your short-term gains for the long-term picture.

If you keep this in focus, you’ll stop your efforts from being bogged down in tiny details that are ultimately irrelevant.

Lesson #7: Drastic changes = drastic results

You’ll eventually hit a point where the low hanging fruit has been picked.

Congratulations, you’re now at a crossroads.

You can either keep trying to provide minuscule improvements to your conversion rate and maybe see some revenue increase.

Or you can start making drastic changes and take a jab at your own explosive growth hacks.

If you’ve hit a wall, consider adding elements you haven’t before, like video.

They’ve had success, with up to 20% increases in conversion rate.

This video from Mint is a great example of an “explainer video.”

Or like when I changed my contact page on Quicksprout into an infographic about my inbox.

These are the types of changes that get you noticed and can shake up your conversion rates when you’ve hit a dead end.

It’s where you can really input your own flair to help take your brand to the next level.


Conversion rate optimization is only hard if you make it hard.

If you can avoid the serious pitfalls that others experience, you’ll be well on your way to improved conversions and revenue in no time.

Just remember that this is a long-term endeavor. There are no overnight success stories here.

Don’t make gut decisions. Your data will beat your gut 99% of the time.

Optimize your site’s performance and take time to ensure your testing software is accurate and reliable.

Don’t fall prey to the quick-fix lure of multivariate testing. One variable at a time is the only way to get actionable feedback.

Above all else, remember that you’re here to improve revenue, not just conversions. The big picture always wins.

As long as you make incremental improvements over time, your conversion rates will turn into multi-million dollar improvements.

What tactics have you seen successfully improve your CRO tests?

Consulting with Neil Patel

See How My Agency Can Drive More Traffic to Your Website

  • SEO - unlock more SEO traffic. See real results.
  • Content Marketing - our team creates epic content that will get shared, get links, and attract traffic.
  • Paid Media - effective paid strategies with clear ROI.

Book a Call

Are You Using Google Ads? Try Our FREE Ads Grader!

Stop wasting money and unlock the hidden potential of your advertising.

  • Discover the power of intentional advertising.
  • Reach your ideal target audience.
  • Maximize ad spend efficiency.
Ads Grader