Beta Lesson 5: A/B Test Player Groups with the New Experiments Tool
overviewCopy link to clipboard
Analytics and A/B testing go together like Link and the master sword. As soon as you combine them, you can bring your A game to the party.
Most game design questions would be easily answered if you could just test your players’ reactions to different solutions. Now you can do just that with our new Experiments feature.
Here’s what you need to know about setting up your first experiment.
1. What is an experiment anyway?Copy link to clipboard
When you create an experiment, you will firstly need to define a set of key-value pairs that will be sent to your players when they open your game.
You can then select players by using a combination of conditions and random allocation, which are defined within an audience group. Experiments are defined by:
- Name – your easy to read identifier
- A key value that is prefixed by ab_ and will be common for all users in the experiment
- A running schedule for the dates the experiment should start and end on
- The timezone for the running schedule
- Audience groups which each have a unique value per experiment they’re sent
2. How do I define my audiences?Copy link to clipboard
You can define your audiences in two steps; one where you add conditions, and one where you define the share of users who will receive the key-value pair (out of the total number of users who fulfil the conditions).
For instance, if you want to test how users with different engagement levels in a certain territory respond to a new type of weapon, you can build two audience groups.
- Audience 1 will include users in the UK with greater than 3 sessions for the past week
- Audience 2 will include those with fewer than 3
However, because this will include all of your users, you should add a rule so that only 30% of users in each group are added to the experiment.
You can also choose to go with one of the steps. As an example, you could choose not to add any conditions, and just send one of the pairs to 30% of all users.
3. How do I see my running experiments?Copy link to clipboard
After you create an experiment, the audience groups will first need to be processed. This may take up to a few hours, during which the experiment status will be ‘Pending’. Once processing has completed, the experiment will become active and players will start to be tagged with the key-value pairs.
The main page will show all of your experiments. This is where you can re-schedule running times, or edit parameters.
4. How do I check the results of my experiment?Copy link to clipboard
Once they are received by players, experiment key-value pairs become game configs. It’s very easy to spot them in the tool because they will automatically show up in the filter picker. You can use it to compare groups within an individual experiment, as well as between users in different experiments.
If you’re not sure which config you should pick, a great place to get started is your experiment list. At the top right of each experiment, you have a ‘view icon’ that will lead you to the overview dashboard with the right configs already selected for that experiment, so you can get insights immediately.
Download SDKCopy link to clipboard
Note: this feature is currently in beta and your SDK must be updated in order to support it. You can download the Beta SDK with supporting documentation here.
more beta tips ↓Copy link to clipboard
We’ve added many more features to the new & improved GameAnalytics with this release. Make sure to check out the other lessons listed below to get the most out of the beta.
- Get Started With the New Beta Dashboards
- Compare Groups of Users Using Basic and Advanced Splits
- Get In-depth Insights Into How Events Drive User Behavior, With the New Explore
- Configure Games Without Coding: Welcome to the Command Center
- Run Experiments and A/B Tests with Your Users, Directly in GameAnalytics
- Defining Audiences for Experiments Using Conditions
- Using the Summary API to Get Quick KPIs
supportCopy link to clipboard
We really appreciate your feedback to help us continue improving GameAnalytics. If you have any comments, just get in touch with our friendly support team using the button below.