UX Walkthrough: Anatomy of a Usability Test in Video Games (Part-1)

Co-authored with Steven Gaston

User experience design has become the mainstay of creating intuitive and rewarding offline, online digital experiences, be it across web, apps physical products or more recently GAMES.

Most top tier gaming companies big or small (be it PC, mobile or console) like Blizzard, Riot, Sony, EA, Zynga, King, Ubisoft, Gameloft understand this scenery change and are now more readily hiring UX talent & integrating UX design in their development pipeline seriously.

I remember finding a big void on this subject a year back and have been writing on the subject of UX design in games under the heading of UX Insights since last year. Analysing F2P games and deconstructing prevailing and upcoming games industry macro, micro trends.

UX Walkthrough?…….What’s it about.

This series takes the subject of UX in games a step further by sharing with you some of the methods and techniques used in the process and demystifying it, I aptly call this series UX Walkthrough.

This is the first article in this series (hopefully one of many!) co-authored with Steven Gaston a very experienced qualitative researcher from market research firm Ipsos.

I and my team members have worked closely with Steve on conducting remote usability tests at their facility, in a clinical setting to get unbiased results and deep player insights, while testing & validating our existing game designs features.

What we will cover in this part?

This part will focus on general assumptions, myths & ways these tests are conducted by many developers, and why they are flawed.

In the next part we will focus on the ideal outline of the methodology employed, learnings, anecdotes and benefits of the process accrued from real world studies.  (Reference to any particular game /feature will be avoided due to confidentiality agreements.)

“It  really doesn’t matter if you are a big and established player of MMO RPG or an Indie working in casual F2P games or RTS space. Usability tests can and should be done irrespective of the scale, they can benefit the final outcomes in a game changing way”

Gaming companies already conduct several forms of testing (QA, Internal play-tests, team sessions, milestone reviews) regularly as part of software development process to identify bugs and milestone build progress internally. On top of these, they also sometimes bring in external players, including friends, family or different team members to test the game. No doubt these tests are immensely beneficial & crucial…

But many developers believe that through these sessions they are also effectively testing for user experience?

… Are they?

Let’s see how TYPICALLY these tests are done and WHY they do not qualify for effectiveUser Experience testing.

Typical example: Team members getting together in a room and play testing.

Typical example: Quick play through with friends, family or other team devs.

Typical UX test.                                                             Why are they Flawed?

• Game played in developer’s office —————— Leads to positive bias towards game.
• Sessions led by company employees —————–Leads to positive bias towards game, as employees working on the game for ages are emotionally invested in the game.
• Asking questions in the wrong manner ————- Leads to game evaluated higher than it should be.
• Asking the wrong questions ————————– leads to game evaluated higher than it should be.
• Survey results as quantitative evidence ————– leads to inaccurate view of the game.
• Research & testing conducted in a group setting ———— People ask each other for help – not representative of how people play in real life, at home or sub-way or at work.
• Testers don’t actually watch or record people play the game ——You miss so much by not having this observation! (In part 2 we will prove why!)

UX testing is just like any form of research.BIASES can very easily come into play and they are not always easy to detect. It takes skill, training and experience to conduct research in a way that ensures theseBIASES have as little impact as possible.

Some classic BIASES that are springing up in our typical test scenarios above :

• Experimenter Bias: This is where the person conducting the research sessions can influence the results in the questions they ask and the way they ask them. An unskilled moderator will inevitably create this bias without intending to. For example:

“how much did you enjoy the game?” may seem like an innocent enough question, but it is asked in a positively leading manner, which may result in a more favourable response.

• Sampling Bias: This is where the selection of people to take part in your research can impact on your results. By only testing on other game developers, your friends or your family, you’re not testing the game with the people who will actually play your game. Ideally, you want to test with people in your target market who don’t design games for a living or know you personally.

People whom you personally know will be heavily biased in their opinion towards your game.

• Response Bias: This is where research participants either consciously or unconsciously feel a need to tell the researcher what they want to hear. This is a bias that takes skill to reduce and will only be increased if you’re doing the testing yourself. A trained external researcher is the best way to get around this one.

An external moderator, not part of the dev. team can significantly help reduce this bias, as his/her framing of questions and observation will not be impacted by previous attachment & history of knowing the game.

• Procedural Bias: This is where the environment in which the research takes place can also influence participants. people can easily get overwhelmed, positively biased & intimidated by a game studios environment, overt hospitality, even things like seeing the creative environment, awesome artwork on walls, devs. and artists at work can put people in awe of the developer, positively biasing their feeling towards the developers & product, even before the session has started!

When you bring people into your office to test a game you’re developing, you’ve once again positively influenced them. Try and conduct your research in a neutral setting.

• Confirmation Bias:  This is where you search for, interpret and remember information that confirms your own beliefs.
• Sunk Cost Fallacy: This is the inability to see potential problems or failures once a venture has been heavily invested in.

We have seen this affect, game designers, artists and even producers who have fought and defended their feature, artwork or UI flow, that were achieved through multiple iterations, & were not willing to budge, but seeing their end users struggle makes them rethink.

• False Consensus Effect: This is when you overestimate the extent to which your beliefs or opinions are typical of those of others.

But isn’t Soft Launch, supposed to bring  forward these issues anyways?… So why bother?

I have heard this argument many a times , where Devs. argue why conduct internal or external usability tests at checkpoints in the dev cycle, you can’t replicate the numbers or data generated from soft launch, & post soft launch you can always tweak the user experience during the polishing phase

Makes sense, EXCEPT Soft launch (though very valuable) is a HIND-SIGHTapproach, done after 80% -90% of your game is developed, code, back-end, games systems, core loop flows, user flows, artwork has been committed and locked. At this stage if you hit a major roadblock in your UX or come across a potentially new insight, that can alter retention and monetisation KPI’s. It’s too late to roll back.

All you can then do now is damage control, push out compromised solutions (many game studios lock down the production schedule months in advance even for updates) or wait for the next big update (2-3 months down the line) while your game is bleeding dry on the rocks, loosing players and the initial launch steam.

Usability Testing on the other hand is a Fore-Sight approach, done correctly at sufficiently regular intervals, even a small group of participants can lead to great cross section insights that can not only help fix existing & anticipate forthcoming problems but give ideas for creating new & engaging features.

In part 2 of this post we will cover, how to set up external usability tests, preparation done at developers (client) end and at research agencies end, how internal and external UX designers working in sync combine their final observation & analysis to produce unbiased results…stay tuned!!

If you liked this post, you can check out my other Game UX Deconstructs