Overview

Most game design questions would be easily answered if you could just test your players’ reactions to different solutions. Now you can do just that with our new A/B Testing feature.

Custom Queries

An A/B test experiment consists of a unique test run for a particular audience during a specific timeframe showing different versions of a game to different groups of players within the audience. You'll be able to compare the different player groups' behaviour when viewing the test results.

Setup and Segmentation

Before you get started with creating and running experiments, please ensure that you have integrated our latest SDKs with your games and have the remote config calls setup correctly. When you create an A/B test, you will first need to define an audience group that you would like to run the experiment on by selecting a combination of:

  • Build
  • Target Percentage
Custom Queries

Your experiment will only be targeted at New Users – existing players will not be included in any A/B tests. Once a new player has entered an A/B test, they will remain exclusively part of that experiment until you stop the experiment.

Create Config and Variants

Now you can define the remote config and variant groups:

- Config Key:

  • Create the key that you want to perform the A/B test on. This key will need to be supported in all the builds previously selected.

- Variants:

  • Each different config value to be tested.
  • You can create up to 3 variants, allowing for 4 test groups in total, including the control group.
  • Players included in the control group, will not receive any value for the a/b test config key.
  • Players will be randomly allocated to the control group or a variant.
Custom Queries

Summary

Here you can:

  • See an overview of the a/b test specification
  • Give your test a title
  • Start the test

In the near future, we will be adding the ability to select further audience filters and a goal metric. For now, these are currently set as “No filters” and “Playtime per session” respectively, but have no impact on the test and the results.

Custom Queries

View Results

You can view data for any test that is Active or Stopped. Some basic values you can see are the experiment length & count of current users in the experiment. We also have some basic metrics available for users that participate in the experiment:

  • ARPDAU
  • ARPPU
  • Conversion
  • Conversion First Time
  • Playtime per Session
  • Playtime per User
  • Ad View (per day)
  • Retention Day 1, 3, 5, 7, 14
Custom Queries

Stop Test

There are several ways to stop a test an active test:

  • Select "Stop" from the overview page actions
  • Custom Queries
  • Select "Stop" from the View Results page
  • Custom Queries