· 5 min read
The Data Export is going away
While we work hard to add more and more features to GameAnalytics, we occasionally have to say goodbye to a feature. Today we’re announcing the well-deserved retirement of our Data Export.
If you currently use our Data Export then please read on. There are some key changes you need to be aware of, and some next steps you may need to take.
Here are the details.
We’re retiring our Data Export
We’re suspending access to the Data Export on May 31st, 2021, a little under three months from now. We’re replacing it with a series of new services we’ve been developing in collaboration with studios and publishers in our community.
- The Player Warehouse is an instant data warehouse that provides access to player-level data via Google BigQuery. We process the data for you, aggregate it at the daily level, and share it in tables so you can instantly query it with SQL. Historical data is available from the start of 2020 onwards.
- The Event Export provides granular event-level data, per event type, with no aggregation, in parquet format. The data is delivered directly to your own AWS S3 or Google Cloud Storage, and can be easily ingested in most data warehouses or distributed databases that support the parquet file format.
- The Raw Export provides all raw data (in JSON format) delivered to your own AWS S3 or Google Cloud Storage. Your data team will probably need to process or transform the data, in order to combine it with other data sources or query it for analysis or machine learning purposes.
Which one should you pick?
The Player Warehouse is by far the easiest and fastest way to access and query your player data, as you won’t need to do any preprocessing or aggregation, and historical data is built-in. Anyone on your team who knows a bit of SQL will be able to analyze this data in a meaningful way, build reports and visualizations.
That said, a higher level of granularity may be required for some use cases. For instance, advanced funnels can be highly dependent on the order players performed certain actions, and you may need individual timestamps to assess when each action occurred. If certain actions in your game have inter-dependencies or are highly time-sensitive, the Event Export may be a better option.
The Event Export delivers each event with its own timestamp, grouped in different files per event type, and including only fields relevant for that specific type. With the parquet file format, the data will be compatible with your existing data warehouse, whether you use a Hadoop-based ecosystem, RedShift, or BigQuery.
The Raw Export is most similar to the existing Data Export and it may be useful if you need the data in a very specific format – either to match the format of a different data source, or a particular advanced use case specific to your game. Processing and transforming data will also require additional infrastructure costs, so unless there is a strong reason you’d prefer raw JSON data, we generally recommend opting for either of the two previous services.
One thing to note about both the Event and Raw Export compared to the current Data Export is that both services will deliver the data to your setup directly. Therefore, your team will save time from having to extract and pipe the data over to your infrastructure, and you won’t have to worry about extra data transfer costs.
Why is the Data Export going away?
As our community of studios has grown, more game teams are looking for access to data in a way that gives them the benefits of a powerful data stack, but without the cost of hiring an entire data engineering team. To make these new use cases possible, we needed to support new data formats, new types of aggregation, and new ways of delivering data to studios.
The tech powering the Data Export was built several years ago and has reached its scalability and customization limits. To develop these new capabilities we’ve had to start from scratch and build a new generation of services, designed for resilience, scalability, and easier maintenance.
This will be a premium service
Please note, these new capabilities are premium data services. We always try to provide like-for-like free capabilities for features we reengineer, such as the GameAnalytics web and mobile apps. But with large-scale, and programmatic data services such as the Data Export, delivery is very resource-intensive from an engineering and infrastructure perspective. As a result, all programmatic access to data is now part of our premium offering.
Although we believe that the value they deliver to studios is worth their price, we have made every possible effort to keep these services as efficient and affordable as possible for studios of all sizes. If you’d like to learn more about our approach to free, freemium, and premium services, we’ve explained it here.
If you’re using the Data Export currently and would like to transition to one of the new options, you can find more details on our Data Services page. The Event Export and Raw Export are now in closed beta, so to access these services or get more information, get in touch with us. We’re also working on more detailed docs, which should be out in the next couple of weeks, along with more information on pricing, and onboarding perks.