How to Be an A/B Test Superstar

We all dream of having a website that converts at astronomical rates—a website that dominates the competition and turns every visitor into a customer. To get there, you have to have a good product or service and a great website. But you also have to be an A/B test superstar.

Okay, so you’ll never get to a 100% conversion rate, but you can almost always increase your revenue or leads by implementing sound A/B testing strategies. Here are three secrets to becoming an A/B test superstar.

Know what you need to test

Before you start running A/B tests, you have to know what you’re going to test. This isn’t a willy-nilly process of simply throwing up variations of this and that. You need a logical approach. If you can’t figure out what to test first, consider these two questions:

1. Where are the biggest opportunities for growth?
2. What elements really matter?

Don’t test something that’s already doing really well when there are other areas that are performing poorly. If your checkout-to-order conversion rate is 75%, then don’t even think about changing it, especially if only 1% of your customers add something to their cart.

Make sure your tests align with your goals. If your goal is to get more people to buy products, then test elements that might directly or indirectly lead to more purchases. That means changing the color of the word “Contact” in the top navigation menu probably isn’t the most useful A/B test, even if someone tells you they think it looks prettier that way. Focus on the elements that are part of the goal cycle. Work backwards from the endpoint and see where you might be losing the most customers. Once you’ve determined those steps, then you can start testing.

Know how you’re going to measure your test

When implementing an A/B test, you need to have all your tracking in place before you set up the test. Don’t think you can set up the test and then figure out how to track and measure later. While you’re trying to sort all that out, you’re going to lose important data that will be vital to determining the winner of your test. To measure properly, you need to know what your success metrics are. If you’re testing a new “Add to Cart” button, then you need to make sure you are tracking the number of times each button is used, the number of times each button appears, and the number of orders placed for each button.

It’s also wise to have audience segments available for analysis. Just because one version is the overall winner doesn’t mean it’s performing better for return visitors or on mobile devices. Be prepared for mixed results, and know how you’re going to handle those. For example, what are you going to do when you find out that your awesome new Add to Cart button results in more cart additions but fewer purchases? Now that’s a dilemma only a true A/B testing superstar can solve. 

Know when to end your test

Ending the A/B test at the right time is an art form. You don’t want to stop it prematurely before you’ve achieved statistical significance, and you don’t want a failed variation to continue running for weeks and weeks. Although A/B testing is a great way to optimize a website, it also has some downsides. The longer you have a losing variation up, the more money you are potentially losing.

The length of an A/B test will vary depending on the amount of traffic and conversions your site sees. Ideally, you’ll need to run your test for at least a few weeks. You’ll also want to reach a 95% confidence level. In most cases, you’ll need at least 200 conversions to make a realistic judgment. It’s just about impossible to declare a winner if one variation had 3 conversions and the other had 4.

Test Your Site to the Next Level

Knowing what to test, how to measure it, and when to end it are essential to running a successful A/B test. If you can do those three things, then you’re bound to find something that works. Don’t be discouraged if your A/B test comes up empty. There’s no shame in going back to the original version. That’s why you’re testing things rather than just implementing changes based on your gut or what you think is working on your competitor’s website.