A quality gate that passed twice and failed the third time
There is a moment when a quality gate stops being a quality gate: the moment a person re-runs it hoping for a different answer.
Our deploy pipeline reached that moment in early August. PilPili is a browser-game arcade of small games, built by one person directing AI for about two hours a day — but this is not a games problem. Any team with a flaky stage in its test suite has met it.
The complaint
Every deploy here is gated. A chain of checks runs before anything ships: the games build, their payloads fit their budgets, declared colour pairs clear a contrast floor, every panel string is read back off a real rendered canvas. Then the two expensive ones: a bot plays every game to its death while a harness measures every piece of text against the pixels actually behind it, and a second harness drives every game's full flow — pause, resume, mute, lose, revive, share, retry. The whole chain took a little over twenty minutes.
One evening a copy change to a single game failed the chain — in a different game, one nobody had touched. Re-run: green. A second deploy later that evening: red again. Same game, same check.
The complaint went into the project log verbatim:
"Why do we run the deploy chain through every game when we publish a single one? I am not waiting twenty to forty minutes to deploy one game when deployment itself becomes a game of chance."
A fair complaint. The pipeline, as it turned out, knew something we didn't.
What the flake actually was
The failing check was the legibility gate. It exists because of an earlier, expensive lesson: when the whole site moved to a warm cream scheme, three games kept drawing their GAME OVER text in the near-white that had been correct against the old dark panels for two months. The words were present, centred, correctly sized — and measured 1.14:1 against the new cream card. Unreadable, and no human had noticed, because nobody re-reads a screen they have seen a hundred times. So the gate does not trust anyone's declared palette. It plays the game, loses on purpose, and measures every string it finds against a screenshot of what is genuinely behind it.
That gate is non-deterministic by construction. It plays. Different runs end in different places, with different things behind the text.
The game it kept failing was our brick-breaker. When we stopped re-running and read the failing measurement, it said this: one hint line on the ready screen measured 2.97:1 against the dimmed field behind it. Our floor is 3:1. Three hundredths under the line — and it had been three hundredths under the line since the day the game shipped. Every earlier pass had sampled a luckier frame.
The gate had not been flaking. It had been telling the truth intermittently, about a defect that was permanent.
The fix took minutes; a comment in the code had all but predicted it. Every other line on that screen already used a light ink, the hint moved to the same one, and the pair now measures 4.70:1. The next day the same class of gate caught the same class of defect in a second game: ready-screen hints in muted brown over a dimmed board, measuring — depending on which bubbles had drifted behind them — as low as 1.00:1. Text that was sometimes literally invisible, on a live game.
The real problem was the blast radius
The trap was structural. The gates that play the games are the best gates we have — each has caught defects that were already live. But because they are non-deterministic, replaying every game on every deploy meant a borderline defect anywhere in the catalogue could fail an unrelated deploy at random. At the catalogue's size then it was an irritation, and every game added raises the odds of a random red — and the natural human response to a randomly-failing gate is the one already on record above: re-run until green.
Re-run-until-green is not a quality process. Once a red gate is treated as bad luck rather than information, its information content is zero, whatever the dashboard says.
The opposite move would have been worse, though. Making the gates deterministic — pinning the bot's path, seeding everything — removes exactly the thing that found the defect. The randomness is the coverage. A deterministic legibility check would have always passed or always failed on one arbitrary frame, and the 2.97:1 hint would still be live today.
So the answer could not be "make it deterministic," and could not be "run everything always." It had to be: run the right scope at the right moment.
Three tiers
The pipeline now has three speeds. The design constraint for all three: no tier may silently skip a check that could catch a defect in the thing being changed.
One game, while iterating. Every gate that can see a single game, run against that game only, including its own bot suite. Under a minute. This is the loop you live in while tuning.
Deploy. All site-level checks, always — nearly two hundred at the time of writing. Per-game gates only for games whose source changed since they last passed. Two to four minutes typically; a deploy that changes nothing gates in about five seconds, down from twenty-three minutes. The copy change that started the argument now gates in under a minute, because it correctly re-checks exactly one game.
The sweep. Everything, every game, the full non-deterministic pass — on a schedule, every morning at six, plus on demand. About twenty-three minutes that nobody sits through, because nobody is watching. This is where the population of runs accumulates, and the population is the only thing that catches the 2.97:1 class at all.
Two implementation details turned out to be load-bearing.
Staleness is computed from source, never from built output. Our build stamps every bundle, so hashing output would mark every game changed on every commit — which quietly restores run-everything-always, the exact behaviour being removed, while looking like an optimisation.
A failed run records nothing. A game is only "known good" if it actually passed. Otherwise a red run would count as "checked recently," and the next deploy would skip the very game that just failed.
The honest price
The tiered gate has a failure mode worth naming before a reader finds it: the fast deploy path is only safe because the sweep runs. The sweep is now the only thing verifying the whole catalogue, and the brick-breaker defect is exactly the class that only it finds. If the scheduled job silently stops — an operating-system update revoking a permission would do it — the fast gate has made the catalogue worse than the slow one ever was, still printing green while sampling nothing.
So the sweep writes a status line with a timestamp and a tag recording whether the scheduler or a human started it. Those are different facts; we were fooled once by a status a human had produced by hand. And because a status line still has to be read, silence now escalates where it cannot be missed: the deploy itself warns loudly once the sweep is a day and a half stale, and refuses to run at all past a week. The failure mode is not gone. It has been moved somewhere it must be seen.
One more change followed from the shape of the incident. A defect at 2.97:1 sat invisible for days because 3:1 is a cliff — pass or fail, nothing in between. The gate now has a warning band: anything under 3.5:1 prints as a warning without failing the build, so borderline cases accumulate visibly, run after run, instead of waiting for an unlucky sample to ambush an unrelated deploy. The band cost a dozen lines.
If this is your pipeline
A test that passes twice and fails the third time is either flaky or telling the truth about a marginal defect, and the two present identically. Read the failing measurement before re-running. Both times we did, the number was real.
If your best checks are non-deterministic because they exercise the real thing, resist pinning them into determinism — fix the scope instead. Small and fast where a human is waiting; everything, on a schedule, where nobody is. Then guard the schedule, because it just became the most important machine you own.
And a re-run in hope of a different answer is worth noticing in yourself. It is the gate saying something, fluently ignored.