· 8 min read

How to Build a Data Warehouse for Games from Scratch

There are six steps to building your own data warehouse: from setting up the analytics code to standardizing the data. Here’s what you can do.

Over our last couple of blogs around data warehouses we’ve explained how they let you analyze data from across your portfolio and look at what insights you can gather from them.

Now, we’re going to dive into how to actually build a data warehouse. What are the steps you need to take and what resources will you need? To figure this out, we’ve rounded up the costs, steps, and tools we think you’ll need to get started. Please note, we haven’t included the cost of running an engineering department (which you’ll definitely need), which can end up being a lot of $$$.

What do I need to get started?

Before you get started, you’ll need to make sure you have the right people. You’ll likely need a software or data engineer, and perhaps an architect or DevOps engineer as well. You’ll also need to budget for the tools you’ll use like data storage, servers and database licences. You might decide to host these on premise or in the cloud. It’s up to you.

Alternatively, you can use our Player Warehouse and let us take the load off your shoulders.

Six steps to your own data warehouse

If you’re looking to create your own data warehouse, there are six steps you’ll need to follow. Make sure you give yourself ample time to finish each step – it’s hard to predict exactly how long each step will take. Some might take three weeks, others three months. Chat with your developer to check.

When it comes to pricing up this entire process, it’s a bit difficult to give you accurate estimates. As you can imagine, there are a lot of moving parts at play. We’ve put some rough costs in this article, but don’t take it for gospel. You can consider it as a rough number that a studio of a semi-large game could expect to pay.

Step one: Add your tracking code

Average cost: $0.
Average time: Anywhere between a week and 6 months.
Typical tools: GameAnalytics core tool.

In this stage, you’re adding the logic to track events in your game that you’ll eventually send to the servers. There’s a lot you’ll want to capture. What are players doing? How long do they stay logged in? When do they make a purchase? What level were they on when they signed out?

Thankfully, you don’t need to plan all that logic yourself. Every analytics platform has its own API you can plug into. Or you can just use our core tool. We’ve already defined all these events and set up the tracking, so you just need to use our API and SDK to grab the information.

If you’re using our core tool, it’ll likely take you a week to set up and learn how to grab the relevant information. But if you’re looking to create your own analytics, this could take a long time. (It’s taken us years to get our core tool to where it is now.)

Step two: Collect the data

Average cost: $2,000 per month.
Average time: One month.
Typical tools: AWS S3, REST API, AWS Kinesis Data Firehose, AWS API Gateway.

Once you’ve got the code in place to collect your games’ data, you need somewhere to send it all and store it. Remember, you’ll need to encrypt the data you send, so any service you use needs to work with TLS – the standard security protocol when sending data over the internet.

You’ll need two services

Sometimes these come bundled together, other times you’ll have to get them separately. The first is a HTTP service at the front, like AWS API gateway. This handles how a device connects to your storage.

The second is your storage, like AWS S3. This is what holds the actual data.

Combining AWS API gateway with S3 is a good mix, as it’ll mean you can use TLS and you can easily scale it up if you get more traffic than you expected.

Consider the cost of the data, storage, and processing power

Whatever service you choose, you need to account for the data transfer costs – specifically your ‘egress costs’. You’ll also need the computer power necessary to handle that much encrypted traffic, the connections themselves, and parsing the events. Calculating this cost can be a bit tricky. It really depends on the volume of your data. You can calculate it here if you’re using AWS Stack (we also wrote a blog on how we reduced the cost of HTTP(S) API’s on AWS. Check it out here).

What other options are there?

A lot of cloud providers offer managed services, taking the load off your hands. For example, AWS Kinesis Date Firehose or API Gateway. These take a lot of the strain off you, but are quite expensive. They’re also not designed specifically for games, so it can be tricky to set them up properly.

Step three: Transform the data

Average cost: $1,000 per month.
Average time: Depends on data size and sources.
Typical tools: AWS Glue, Hadoop (EMR), Kafka, Kinesis.

Now that you’ve collected all that data, you need to convert it all into a standardized format. The idea here is to process the data so that you can add it to the warehouse. This stage is crucial if you want to bring together multiple data sets from different sources, otherwise, you won’t be able to search through the data properly or find links between those data sets.

Depending on how you’ve set up your original databases and how many different sources you’re gathering from, this could be relatively simple or quite complex. Lots of sources with different rules and data sets are going to be difficult to standardize.

Usually, you’ll be processing in batches, using AWS Glue or Spark on Hadoop (EMR). But in some cases, you might want real-time data. In those cases, you’ll want a streaming service like Kafka or Kinesis.

A word on pricing

These steps are usually quite pricey. It really depends on the scale of your solution. Post-processing data alone (adding the cost of the transform) could get up to $12K, and then if you add the cost of the player ETL, this can be around $5K per month.

Depending on how much data you’re dealing with (so say billions of events), it can cost you tens of thousands of dollars each month.

Step four: Load data to the warehouse

Average cost: Anywhere between $100 and $2,000 a month.
Average time: Ongoing. Usually once a day.
Typical tools: BigQuery Data Transfer, AWS Glue, Apache Druid, or Clickhouse.

Once you’ve prepared all your data and it’s in a standard format, it’s time to send it to the warehouse.

Rather than sending it all at once, it’s best to send a batch once a day. You’ll also want to manage how much you send: base this on your typical queries and the data lifecycle. This also means you can keep a handle on the costs and not send too much at once.

The most popular warehouses are Snowflake, BigQuery or AWS Redshift. And if you need real-time data, you’ll want to look into Apache Druid, Apache Pinot or Clickhouse.

Step five: Monitor and troubleshoot

Average cost: Free (or not too pricey).
Average time: Around a week.
Typical tools: AWS, Cloudwatch, Datadog, Grafana, Pingdom or Pagerduty.

Once everything is set up and the devices are sending data, it’s time to make sure it stays online. Any downtime is going to lose you data, which could be crucial.

You’ll need someone on call to deal with problems, day and night. Usually, a small team of two or three should be able to handle this, along with their other responsibilities. But bear in mind that they’ll need to check the status regularly and have the skills to fix any problems on their own.

Step six: Analyze and visualize

Average cost: Free (although can be thousands, depends on what you need).
Average time: At least a couple of weeks.
Typical tools: Tableau, Superset, Holistics, Looker, Google Studio, or AWS QuickSight.

Finally, you need to make that data useful. If you’ve got someone who knows SQL, they can run queries. But this often needs specialist knowledge. Usually, studios go for a visualization tool or framework, like Tableau, Superset, Holistics, Looker, or AWS QuickSight.

Our Player Warehouse is ready to go

Instead of all that faff, you could just use our Player Warehouse. We’ve designed it specifically for game developers. And you can get started in minutes, not months. Let us handle keeping it up and running. And save yourself all the hosting and processing costs.

That way you can focus on the last step: actually analyzing the data. So get started and make the most of your games.

Get game industry insights,
delivered to your inbox

Announcing HyperBid 🎉

New mediation platform. Built for games.

Sign up to get free access to GameAnalytics premium products.