A/B Testing is currently in early access. You apply to the early access program please go here.
Analytics and A/B testing go together like Link and the master sword. As soon as you combine them, you can bring your A game to the party.
Most game design questions would be easily answered if you could just test your players’ reactions to different solutions. Now you can do just that with our new A/B Testing feature.
Here’s what you need to know about setting up your first experiment.
Technical IntegrationCopy link to clipboard
Before you get started with creating and running experiments, please ensure that you have integrated our latest SDKs with your games and have the remote config calls setup correctly.
You can find out exactly how-to here.
1. Setup and SegmentationCopy link to clipboard
When you create an A/B test, you will first need to define an audience group that you would like to run the experiment on by selecting a combination of:
- Target Percentage
Your experiment will only be targeted at New Users – existing players will not be included in any A/B tests. Once a new player has entered an A/B test, they will remain exclusively part of that experiment until you stop the experiment.
2. Create Config and VariantsCopy link to clipboard
Now you can define the remote config and variant groups:
- Config Key:
- create the key that you want to perform the A/B test on. This key will need to be supported in all the builds previously selected.
- Each different config value to be tested.
- You can create up to 3 variants, allowing for 4 test groups in total, including the control group.
- Players included in the control group, will not receive any value for the a/b test config key.
- Players will be randomly allocated to the control group or a variant.
3. SummaryCopy link to clipboard
Here you can:
- See an overview of the a/b test specification
- Give your test a title
- Start the test
In the near future, we will be adding the ability to select further audience filters and a goal metric. For now, these are currently set as “No filters” and “Playtime per session” respectively, but have no impact on the test and the results.
4. View ResultsCopy link to clipboard
You can view data for any test that is Active or Stopped:
- Experiment length
- Count of users currently in the experiment
- Metric comparison table
- Metrics for each Variant, compared to the Control Group.
- For each metric, we calculate the mean for the duration of the experiment and update
- The table is updated daily
- Conversion First Time
- Playtime per Session
- Playtime per User
- Ad View (per day)
- Retention Day 1, 3, 5, 7, 14
5. Stop TestCopy link to clipboard
There are several ways to stop a test an active test:
- Select “Stop” from the overview page actions
- Select “Stop” from the View Results page