Docs > Knowledge base > Features > A/B Testing

A/B Testing

Early Access

A/B Testing is currently in early access. You apply to the early access program please go here.

Analytics and A/B testing go together like Link and the master sword. As soon as you combine them, you can bring your A game to the party.

Most game design questions would be easily answered if you could just test your players’ reactions to different solutions. Now you can do just that with our new A/B Testing feature.

Here’s what you need to know about setting up your first experiment.

#

Technical Integration

Copy link to clipboard

Before you get started with creating and running experiments, please ensure that you have integrated our latest SDKs with your games and have the remote config calls setup correctly.

You can find out exactly how-to here.

#

1. Setup and Segmentation

Copy link to clipboard

When you create an A/B test, you will first need to define an audience group that you would like to run the experiment on by selecting a combination of:

  • Build
  • Target Percentage

Your experiment will only be targeted at New Users – existing players will not be included in any A/B tests. Once a new player has entered an A/B test, they will remain exclusively part of that experiment until you stop the experiment.

#

2. Create Config and Variants

Copy link to clipboard

Now you can define the remote config and variant groups:

  • Config Key:
    • create the key that you want to perform the A/B test on. This key will need to be supported in all the builds previously selected.
  • Variants:
    • Each different config value to be tested.
    • You can create up to 3 variants, allowing for 4 test groups in total, including the control group.
    • Players included in the control group, will not receive any value for the a/b test config key.
    • Players will be randomly allocated to the control group or a variant.

 

#

3. Summary

Copy link to clipboard

Here you can:

  • See an overview of the a/b test specification
  • Give your test a title
  • Start the test

In the near future, we will be adding the ability to select further audience filters and a goal metric. For now, these are currently set as “No filters” and “Playtime per session” respectively, but have no impact on the test and the results.

 

 

#

4. View Results

Copy link to clipboard

You can view data for any test that is Active or Stopped:

  • Experiment length
  • Count of users currently in the experiment
  • Metric comparison table
    • Metrics for each Variant, compared to the Control Group.
    • For each metric, we calculate the mean for the duration of the experiment and update
    • The table is updated daily
      • ARPDAU
      • ARPPU
      • Conversion
      • Conversion First Time
      • Playtime per Session
      • Playtime per User
      • Ad View (per day)
      • Retention Day 1, 3, 5, 7, 14

#

5. Stop Test

Copy link to clipboard

There are several ways to stop a test an active test:

  • Select “Stop” from the overview page actions

  • Select “Stop” from the View Results page

 

 

Contact support

We’ll get back to you ASAP.

Report a bug

Let us know more details.