The perfect choke point, and the caller that broke it
Honest status, up front: the analytics described here is designed and built to the point of sending, and deliberately not yet live — that is part of the story.
PilPili is a browser-game arcade of small games, built by one person directing AI for about two hours a day. Most of the catalogue keeps score. This month we designed the analytics that will tell us what players actually do in those games — and the most instructive part was a single wrong assumption that looked, right up until we read the code, like the cleanest piece of the whole design.
The question the analytics must answer
Store-bought analytics tells you sessions and page views. For games we need one harder thing: when a run ends, what was the score? The distribution of end-of-run scores is where every design question we care about lives — is there a long tail of high scores, does the difficulty ramp kill everyone at the same wall, does a change to one game move its curve. Get that event wrong and every chart drawn from it is confidently misleading.
Two design decisions came out of that one requirement, both worth stating because they cut against defaults:
We rejected a managed analytics product whose write path samples. The documentation said, plainly, that sampling can occur on write and on read. Averages survive sampling; tail counts do not — and "is there a long tail of high scores" is precisely a tail count. A tool that quietly thins your data is fine for traffic dashboards and wrong for distributions.
We batch events in small flushes, not big ones. The natural engineering instinct is a large batch — fewer requests. But an unflushed batch dies with the tab, so loss scales with session length, and long sessions are where the high scores are. A big batch samples away your best players; it's the same failure as the managed product, self-inflicted.
The perfect choke point
Now the instructive mistake. Every score-keeping game in the catalogue already calls one shared function at the moment a best score might need recording. One function, every scoring game, already wired, already tested. Emit the end-of-run event from there and no individual game needs touching — the single most elegant integration point I have ever not shipped.
Before wiring it, we read every call site. Every game but one calls that function exactly once, when the run ends.
The last one calls it on every move.
Our merge-puzzle game banks the player's best mid-run, deliberately, so someone who closes the tab halfway through a board keeps the best they reached. That is correct behaviour for a game you put down and come back to — it must not be "fixed." But emit the end-of-run event from that shared function, and this one game fires it hundreds of times per game, pouring every intermediate score into the distribution. The chart would not look broken. It would look like a game with a suspiciously smooth score curve and enormous play volume, and every conclusion drawn from it would be wrong.
So the event is emitted where a run actually ends — one engine-level call covers most of the catalogue, and the oldest games that still hand-roll their own game-over each emit it themselves. The daily puzzles emit nothing, correctly: they have no scored run.
The subtler lesson: some games can't share an axis
Reading those call sites surfaced a fact about the data that no amount of pipeline correctness would have: the merge game has no terminal score for abandoned runs at all. A player who quits mid-board simply stops; nothing fires. So its score distribution is a distribution of completed runs, while every other game's is a distribution of all runs — and putting them on the same chart axis is a category error dressed as a comparison. That caveat is now written into the project record next to the design, because the person reading the chart in six months will not remember the call sites.
Why it still isn't live
The pipeline is built to the point of sending and deliberately stopped there, for reasons that are themselves choke-point lessons:
The event beacon must compile to nothing in builds destined for game portals — distribution partners review submissions, and a bundle phoning an unknown host is exactly what a reviewer should flag. So the beacon exists only in builds for our own site and app, not as a runtime switch but at compile time: the portal build contains no external call to find.
And the events schema is an allowlist, not a denylist — a new key must be deliberately added, because analytics grows by accretion and every "just one more field" is a future privacy answer someone has to defend.
What I'd hand you from this
Decide what question your data must answer before choosing tools, then test the tool against the question's worst case — for us, tail counts against sampling. The default tool was fine for the average and wrong for the exact number we built all of this to learn.
Write the data caveats down where the data lives. "These two distributions are not the same population" is obvious the week you learn it and gone within a month. The chart will outlive the memory.