Why Painted Door Tests are a Secret Weapon in Game Development

Have you ever launched a new feature that you were confident would succeed and be game-changing? Not only did your gut tell you it was a good idea, but you were able to back it up with user research?

Then, when you finally launched, you found the new feature made very little difference. Perhaps users didn’t notice it, or they just didn’t care about it in actual game play. All the time, money, and resources you put into it could have been spent elsewhere.

This has happened to me countless times, and it’s why we at Solitaired now take a data-driven approach towards understanding what new features we should work on.

A $1M mistake

I learned the hard way that both my instinct and consumer research could be wrong. Before getting into the gaming space, I had run an educational business where one of our products would help students create citations and references.

We had collected over 1 billion citations, and because they reflected the sources students used in their research, we thought we could turn that data into a unique search engine. We spent $1M over the course of a year doing user research and building a one of a kind search engine. We were confident it would be a hit.

Then when introducing the feature, we found that hardly any of our users clicked the navigation link to use it. It turns out students didn’t need it, and there was a great product called Google (heard of it?) that already helped them with their research.

Instead of putting all this energy into building a search engine, what if we displayed a button to the not-yet-built search engine, and measured how many people clicked it? With a fake feature test like that, where we are “painting a door”, we would have learned it was unlikely to be successful from the get-go. And that would have saved us a lot of time and money.

Running painted door tests on our gaming site

When we launched our new gaming platform Solitaired (which connects classic games to brain training), we had this lesson in mind. Instead of investing in doing consumer research with focus groups and wireframes, we decided to run some painted door tests.

Test one: hint feature

Our first test was with a hint feature. We hypothesized that many of our users didn’t fully understand how to play some of our solitaire games, and a hint feature could guide them through the game.

We introduced a button that said “Hint” to 20% of our audience, where when clicked on we’d explain the feature was coming soon. We found that over 15% of the users in this test group would click the hint button.

Knowing that, we decided to spend two weeks building out the feature. As a result, this feature has improved completed games by 10%!

Test two: multiplayer feature

As another test, we hypothesized our users may want a multiplayer feature, where they would race other people to complete a solitaire game. Building a feature like this would be a considerable undertaking, and it would have to make a meaningful improvement to our user engagement metrics to be worthwhile.

Instead of building it out, we introduced a button in our navigation and a new game menu to encourage multiplayer usage. Perhaps not too surprising, for a game meant for solitary play, less than 2% of our user base clicked on the button. It was a clear signal that we shouldn’t invest our time in multiplayer.

Test three: new games

While you might be familiar with the classic version of solitaire, there are other popular variations like FreeCell and Spider. We tested if our userbase wanted these games by putting links to them in the navigation. When clicked we had a pop-up that said the games were coming soon. 5% of our users clicked to play, which was material enough for us to build the games. Now our Freecell and Spider games make up 10% of our usage.

How to best run painted door tests

While running painted door tests has improved how we spend our time, it doesn’t work for all product ideas.

As part of the roadmapping process, we identify initiatives, their impact, and whether we can run painted door tests on them. If we believe the initiative has potential for high impact, and it’s something where we can run a painted door test, we’ll prioritize it.

Here are some considerations when running a painted door test:

1. You need a large number of users

Let’s say you launch a game, but in the first few months, you’ve only amassed a few hundred users. If you run a painted door test, there is a chance that the results of the test may not be statistically significant. This means if this test failed or succeeded, it’s possible if you run the test again you can get different results.

You can either run the test longer until you reach statistical significance, or wait until you have enough users where you can get quick answers from painted door tests. If you have 10,000 players monthly, for example, you can get answers and data much faster.

2. You have to give the test a fighting chance

Let’s say you want to run a test on a leaderboard feature for your game. However, the link to the test is buried in your sub-navigation, where users don’t look.

In this instance, you may be setting yourself up for failure. Users might not find the button to access the leaderboard, whereas if you placed the feature link in a more prominent location, you could have gotten much better engagement. Therefore, you have to be thoughtful on how you design the painted door test.

3. Sometimes you don’t need to create a painted door test

With new features, you have to judge how much effort the painted door test will take versus building the feature. In some instances, running a painted door test is a no brainer, and in others it might make sense to spend a little extra time and develop the feature.

For example, when it came to adding a timer on our solitaire site, it was just as easy to add the feature as run a painted door test for it.

4. Is there room for optimization?

If a painted door test fails, it doesn’t necessarily mean the concept your testing won’t work. Any good product team will know that success comes with iteration. Use your judgment to determine if you should continue to invest in an idea, even if a painted door test fails.

For example, we ran a painted door test on a subscription offering, and found that hardly anyone would subscribe. Nevertheless, we believed there would be room to constantly improve the subscription offering and went ahead building out a subscription plan. While our conversion rate was initially consistent with the painted door test, through optimization, we’ve already doubled it. Moreover, we believe there is plenty of room to further improve it.

5. Some features take time

Painted door tests are excellent for getting an initial read on feature uptake. However, it won’t tell you how your users will use that feature over time.

A feature like trophies may not show any improvement to KPIs upon an initial painted door test, however, over time it may drive more and more returning users and engagement.

You have to evaluate if a painted door test can give you a long term signal on feature performance. If it can’t, go through your standard user development processes to assess if a feature is worth building.

6. Don’t run multiple painted door tests

You may be used to running multiple tests at a time. With painted door tests, limit it to one at a time. If users come across numerous calls to action to features that don’t exist, that can create a poor user experience. Moreover, it may conflate your data, as one test may impact the other and it will be hard for you to determine why it’s happening.

Give it a shot

While not a perfect solution in all situations, painted door tests are incredibly useful for game development. You can get a quick read on what your customers really care for, and better spend your time on what truly moves the needle. If you have any questions, feel free to get in touch with us here.