SC2 (RTS) Data Manager

Context

I used to play quite a lot of real-time strategy (RTS) games competitively and I always loved collecting various data and crafting summary tables, rankings etc just for fun. This project combines the two passions, as it is primarily used to gather and store leaderboard data from games I was interested in at some point and I reuse the data in other projects.

What I Built

This project is a collection of various crawlers and data API connectors that are intended to run repeatedly to collect daily snapshots of leaderboards. Since the APIs rarely offer a peek at historical data, this is the way how I can track and compute statistics like daily activity, ranking shifts, activity bursts, long-term performance, etc. With the data collection, I can reproduce the calculations and I can even retrospectively alter or add new ways to process the data.

During the project's lifespan, I've integrated various sources and APIs such as Battle.net / Blizzard community API (StarCraft 2 data), SpireGG, AlphaTL (tournament platforms), AoE4 leaderboard, AoE4 World, Twitch (stream tracking), Untapped.gg (Stormgate), web scraping.

The data was stored initially in an RDF store (Jena Fuseki) as that was the technology I wanted to explore and understand more deeply. Later I started storing the data in the more commonly used PostgreSQL database.

There are various utility scripts that wrangle and reshape the data and transform it into outputs relevant to my interests or based on community requests. I worked in an international hobby game-analyst group, with local semipro/pro teams (eSuba, Inaequalis) and local amateur tournament organizers (OvaChamp, ESOgames).

Some of the scripts are not glamorous, but they just served the purpose I needed: take a spreadsheet from a tournament, scrape or import data from a platform that may not exist next year, turn it into Turtle/CSV/XLSX, and make it usable by the webapp, Discord reports, or later analysis.

Interesting Problems

• Feature creep, extending scope — the first iteration dealt with SC2 leaderboard data and was only focused on known CZ and SK players, so I could provide abridged leaderboard for the local community (which was a feature missing in the SC2 client). It only provided the current snapshot. Eventually I got to dealing with data from all regional servers; tracking historical data; grabbing player profile data; extending to other trending RTSs; adding community-driven views and exports; Discord reporting

• Instability of SC2 endpoints — the APIs were never the main priority for Blizzard (SC2 game developer) and they frequently served stale data or were inaccessible. At some point I had to incorporate a triple fallback mechanism to guarantee I get at least partial data (the flow is still used by the premier web ladder tracker).

• Niche community position — when I started the project, I was rather new to developing and doing public-facing work, so I decided to lessen the ambition and primarily provide for the Czech community I was part of instead of trying to claim a global userbase. I also suffered a bit of impostor syndrome, or rather I was not confident enough to approach people working on other cool SC2 projects. This somewhat nerfed the influence and position I gained, but at the same time I could slowly work in the shadows and learn to provide reliable stuff for other users without too much pressure.

SC2 data state quirks

As mentioned above, the data acquisition is not nice and straightforward. For context, there are ladder seasons spanning ~4 months and the players are divided into groups (ladders) of up to 100 players and the ladders are further separated into divisions based on skill percentile (older article describing the concept).

In the ideal world, the loop would be:

• for each division [A] get list of ladders in the current season
• for each ladder [B] get the current stats
• consolidate

Over time, there were cases when [A] only provided a list of ladders known at an older timestamp, or it even provided an empty result. Similarly, [B] sometimes got stuck at historical data instead of serving actual states that could be seen in the in-game client.

To deal with the first issue, I started utilizing the fact that ladder_id values are incremental and was just shooting forward during each snapshot download to see if there are any unknown ladders to discover. The ID sequence was mostly without skips, so the generous rule of 5 consecutive IDs not existing means we got everything. I kept the list of IDs in the database, so I don't waste API queries.

The latter issue was more complicated to deal with. There was a newly introduced set of API calls to get the ladder data [C1, C2]. C1 only needed the ladder_id, but provided limited info (missing rating info for example). C2 had richer info, but it was on a player profile endpoint, so I needed to query the player via player_id and choose the particular ladder_id they are a member of. This forced me to first grab the useless C1 just to learn who is in the ladder group so I can call C2 with proper input.

To make matters more complicated, even C1, C2 served stale data on occasion (not correlated with B functionality). So I discovered there is actually a player profile that can be accessed from the web and web-crawled the data [D] as a last resort.

So we ended up with:

• try to get list of ladders [A]
• try to explore ladder_id that are after the maximum known ID
• merge the found ladder_id and for each:
  • run [B] and [C1+C2] (with retries)
  • if nothing is found, try [D] (without retries)
• consolidate

To put this into perspective, there are about ~650 ladders for Europe server and 1v1 game mode. When I collect all the regions and game modes, I deal with ~2900 ladders per snapshot. Each ladder costs 3 API calls in the best case.

Related projects

This project is the backbone for my other projects and analytics such as:

• SC2 Tour System - public-facing tournament and community operations site that uses this data layer for ladders, players, and RTS community views.
• RTS analytics reports (Stormgate 2024-02, Battle Aces 2024-07) - longer data-analysis writeups that reuse the same habit of collecting imperfect RTS data and turning it into community-readable observations.
• SC2 Streamer Companion & Win Chance Predictor - local replay-monitoring and ML side quest that sits closer to live play, opponent history, and win-chance experimentation.
• SC2 Enemy Info - older desktop helper that queried the SPARQL-backed player data during live SC2 matches.
• Daybreaker - Discord bot that mixed Battle.net data, replay parsing, charts, and community commands.