Navigating the hazards of game data

I’m a bit of a dataphile at heart and I strongly believe in the importance of using real-user data within game design; in fact I’d suggest effective data capture is vital to delivering games as a service. The trouble is how do we know what the data means? How can we be sure we are asking the right questions?

Live Data capture is still relatively new to the game industry and it’s often treated as a mythical art; something only ‘genius’ data analysts can understand. Don’t get me wrong, these guys are a great asset (especially the genius ones) to make sure the insight you get is meaningful, but only the exceptional few will think like a game designer.

First thing we need to know is what questions we want to ask. Sounds easy doesn’t it? We usually want to know whether people like our game, whether they will spend money on it and how we can keep them playing. But we can’t answer these with data and when we ask focus groups qualitative questions about the why and how of their behaviour it’s inevitably influenced by the questions themselves. Asking people what they like and what causes them to stop playing my game is useful; but I need to measure the significance of the response of the people I asked. For that we need to measure what players actually do.

Obtaining the right data

Data capture can only tell you when an event is triggered. How many people enter a scene, for example. An event is a positive action which can be recorded by the client software or the server. It usually has some parameters associated with it. For example which player, what game session, what date/time, how much health they have, what score, which scene, which spawn point.

There may also be a range of data points we don’t need to capture, such as the layout of the scene because it doesn’t change or it can be inferred from other data. But what events to capture? That’s going to depend on a number of factors. How much budget do you have for servers, storage, bandwidth and data analysis software? What level of detail is going to really be useful? Does every footstep count or is just an XYZ location when an action occurs sufficient?

For me, there is a technique called Hazard Analysis and Critical Control Points or HACCP which I find useful to help me think about the kinds of data I want to capture. It’s a technique for Food Safety (which I learned about in my first job working on software for the food industry) developed by Pilsbury, NASA and US Army Labs to provide food for Apollo missions. It’s a testing model which works on the principle of understanding where in a process is the critically relevant point to conduct a test against a hazard.

Step 1 is to understand the potential hazards and how these might be prevented. For games (particularly as services) we want to understand where players leave the game and don’t come back. So we have to understand where the players might drop out or become frustrated and what actions might trigger a return to play.

Walking through the first time user experience can help us understand some of those points. Mapping that process with events so we can detect players leaving the game and later returning to play is a good start. But perhaps we need to know more about the conditions they experience prior to that departure.

These events we are identifying are our Critical Control Points and identifying them is Step 2. The problem with games, particularly client/server applications, is where you can get the best data, especially if you want your game to be playable offline. The problem is that you can’t usually get an ‘End’ event. This was a problem we had to deal with when setting up the data capture for PlayStation Home.

If a Player pressed the PSButton on their controller and they left the experience and there would be no more data posted to the server, we only knew that they left because the server disconnected and we couldn’t know if that was voluntary or an external problem. Therefore you have to work out what data you can collect and capture that. For example, to overcome the problem we had at PSHome we stored historic data locally and posted cached data when the player returned to the service; which has its own complications.

Thinking through the ‘first time’ experience really helps and make sure that you capture the selection, start and finish of each level/scene of your game as well as critical actions during play such as an interaction with an object, combat, taking damage, etcetera. Where these stop can tell you a lot about what’s going on.

Understanding player behaviour

Step 3 is identifying the critical limits for each control point. To do this we have to understand what normal behaviour looks like. This isn’t the same as an average. Different segments of players may exhibit different ‘normal behaviours’. This process of identifying limits is important to assess the significance of any changes and in games we will have to constantly review these limits based on real-world players. There are lots of things to consider when assessing what is normal, including statistical significance and correlation vs causation, but that’s where it pays to consult your friendly data analyst.

Step 4 is where we actually set up the monitoring process and where we define the events, parameters, database entries and reference data which allows us to make sense of our reports. The reporting formats are important and I find funnel reports showing the percentage of players who get through each stage of the game and continue to the next to be the most useful. There is of course value in measuring the business data such as ARPU, ARPDAU, K-Factor, etcetera, but the Churn (loss of users) between each step of the experience has always been the most useful to me.

I always recommend that you compare players going through similar stages in their lifecycle, at the very least try tracking people based on their progress in terms of number of days of actual play rather than the actual date when they downloaded the game. There are more sophisticated ways to do Cohort analysis, even A/B testing, but I think that’s a topic for another article.

In Step 5 we establish corrective procedures and this means including time in your post-release sprints to fix problems identified through the data you capture; and as quickly as possible. Avoid interrupting your current sprint unless there is a genuinely show-stopping bug. Give yourself time to be sure that you understand the data and have a good solution before acting.

It’s easy to assume that the problem is at the point where the data goes bad; sometimes the problem is earlier and it’s often not what your most vocal players complain about. It’s not just negative data we are looking for. Look for positive moments in play too. Where are players sharing their achievements or posting gameplay videos to the network? Thinking about what is happening at all points of interest allows us to focus our attention on what matters to those players. It’s our job to try to understand them and identify how to accentuate the positive and eliminate the negative.

Steps 6 and 7 are about reviewing the process itself and making sure we have a record keeping process respectively. These are important as we have to make sure what we learn from creating our data capture process and what we identify through it about how players interact with our game is vital to our long term future. Especially if we want to avoid repeating our mistakes.

Data capture has to answer design questions for it to have value in our industry; just looking at the commercial performance can lead us down the wrong path and diminish the lifetime value of our game. A game is about entertainment and a dialogue with players; as a designer you need to understand what your players want and how can you do that if you don’t have the senses to listen to them.