Trivial vs. Radical A/B testing

It’s relatively easy to have a high percentage of winning tests in A/B testing.

But it’s better to run the kind of tests that have a smaller chance to win.

Here’s why.

How to have a high percentage of winning A/B tests

First, let me tell you how to have a “successful” A/B testing system.

If your main KPI is to have many winning and your career depends on it, just do this:

• Run simple tests
• Many of them
• Tests like:
• Button colors
• Moving elements on the page
• Adding a few words to CTAs
• For each test, have more than 3 variations
• This will maximize your chance for success
• With many variations for each test, you’re almost sure to hit a variation that will be more effective than your control
• Even if not, there’s always a chance of a false positive, when one of your variations is not better but it still wins purely by chance and randomness

That’s it, you’re now running a “successful A/B testing program” and you can boast 85% winning tests.

Why would you want fewer wins?

The method above works but it’s not exactly doing much for the business. Overall, long time conversion rate is probably completely flat, maybe even falling. You’re basically standing still.

What you can do instead is swing for the fences.

Run bold tests. Tests that make you cringe a little when you hit “Start” in your VWO, Optimizely or Google Optimize. Tests that business owners reject when they see them and don’t even want to test. Convince them to run your crazy ideas. Force them, if you have to.

Why?

While your overall percentage of winning tests with bold ideas will be significantly lower, the winning tests will more than make up for it.

In this hypothetical example, you’d have half the winning tests compared to the trivial testing. But the business would definitely feel it, both times. The needle would move and would stay at a higher level.

Radical tests can be about:

• Positioning (potentially the most promising and least used type of test)
• Rewriting copy
• Redoing the whole funnel
• Reducing scope of the product or service
• Removing many elements, instead of adding

Conclusion

Stop playing with colors. Start playing with crazy ideas. Maybe you’ll need to start with a few trivial tests to show some green in your results but then go for radical, even if your next test fails badly. Even if your next three tests bomb. The forth may be a game-changer for your business.

No button color test has ever been a game-changer for any business.

PS. It’s hard to persevere through losing tests. It makes you look like you don’t know what you’re doing. Until you hit the jackpot and all the losses are forgotten. You don’t judge the winners.