5 A/B tests to run during your soft launch

As we released A/B testing last month, we thought we’d share a few tips, tricks and ideas on how to make the most of your experiments. So put on your lab coat, fire up the flux capacitor and let’s get to sciencing.

If you’re new to A/B testing, it’s a pretty simple concept. Send three variations of your game out into the wild: A, B and a control group. One group of your users sees A, another sees B and the last sees no change at all (your control group). You pit them against each other and see which performs better. It’s survival of the fittest in action.

But when should you start your A/B tests? And what exactly should you be testing?

Your soft launch is the Goldilocks zone

Before you can start testing, you need players. And the best place to find them is during your soft launch (you’ll probably still be stamping out bugs before then anyway). And you don’t want to wait until you’re live to start testing out ideas.

That window where the game is almost complete, but you haven’t made an official announcement or invested a huge amount into marketing: that’s the perfect time to test. You can be wacky and creative, trying out ideas and scraping the ones that don’t work.

It’s the time to see if your hunches are right. To take a punt and let the best idea win. To decide on those fiddly choices that nobody’s quite sure about, like deciding on which image to show while your game is starting or updating.

1. Test your ads

We’ve all loaded up a game and been bombarded with adverts, jamming up the flow until you just rage quit and uninstall. And we know to avoid it. But you still need to show ads (if that’s your model). The trick is finding that balance: enough to make revenue but not so many that they become frustrating.

So how often should you play them? Every level? Every five? What kinds of ads should they be? A video that gives the player a reward or a banner across the bottom of the screen? And, if you’re using banners, where should they go? Bottom? Top? Along the side? Slap-bang in the middle?

2. Test out your prices

There are two sides to this. Your in-game economy and your real-world prices.

When it comes to the real world, it’s relatively simple. How much are people willing to spend on a new pack, some gems or a cosmetic item? Is 99 cents enough or is it too high? What if you offer bundles? How much of a discount works as the ideal encouragement? What’s that cut-off point between the offer being a great deal, and it feeling like the original price is too high? (If you hit that, you might find players refuse to even buy the discounted bundle out of protest.)

But you can also test ways to improve your gameplay with your in-game economy. For example, we’ve all saved our best potions for the end boss – only to never actually need them. Can you encourage players to use those items more if you lower the price? What if you changed the cost to be more like mana points that recharge?

Lastly, you should think about your virtual currency: that crossover between the real and virtual. How fast should you give players a currency that they can also buy with real-world cash? You don’t want players to feel like your game is pay-to-win, but if you give the currency too quickly there might not be any incentive to buy it at all.

Whether it’s a virtual currency, an in-game economy or specific items, subtle changes in the cost can have a big impact.

3. Test out the difficulty of your tutorial

When you run an A/B test, it’ll only run for new players. So the most important aspect is your tutorial. You’re unlikely to see much difference if you try out two variants that affect the color of gems which only appear half-way through the game.

Instead, focus on those early levels. If you give too many rewards too quickly, they’re going to lose value and your players will grow bored. But make it too hard, or don’t reward them enough, and they’re just going to rage quit.

The trick is to spot a pattern. Is there a big spike of players dropping off after around an hour of playtime? Then grab a friend, watch them play through and see what happens when they’re an hour into the game. Perhaps they spend five minutes bumping against a problem and you just need to add in a clue. Perhaps it’s just that the last fifteen minutes have been too repetitive.

It’s worth setting up and A/B test that affects the difficulty across all your tutorial levels, and also testing specific levels. For example, imagine you’re testing the speed of a ball. One test might affect every single level: is it better faster or slower? But what if it’s just a specific level that’s particularly tricky? To optimize each level, you’ll want individual tests.

4. Test your aesthetics

There’ll always be those choices you just can’t decide between. Sometimes it’s small, like choosing between whether a button should be green or blue. But other times it’s a bit bigger. Maybe you’re deciding which of your gems should be the most ‘valuable’ and give the player the most points. You can often be surprised at how a simple change can ripple out.

And if you have the time, and a team with a bit of patience, you can even consider testing your characters. Say you have a side quest you want people to join. Do more people talk to the quest-giver if it’s a mysterious orc or a fabulous high-elf?

5. Test your store’s layout

Lastly, it’s a good idea to experiment with how you actually display your items and bundles in your store. You can change the order, the size and even the color and pizazz surrounding them to see which approach entices players the most.

Obviously, there are common practices to follow. (You probably want your menu at the top left of the screen, for example.) But even those elements are guidelines. Research what other games have done and see whether it might be appropriate for you.

That’s the whole point of an experiment: to see whether it’s right for your game and your players.

Start your experiments

Whether you’re testing your ads, your prices, your difficulty, your aesthetics or your layout: you don’t need to leave it to guesswork.

So if your safety goggles are safely strapped on, a fire extinguisher is at hand, and you have an idea you want to test out, give our A/B testing tool a try. It’s free and you can run up to ten experiments at once. Also we’ve now released some new updates and improved our A/B Testing tool. Head here to get the latest news.

Just remember not to feed it after midnight or get it wet.