Thousands of games, zero frames

You cannot settle a design argument by playing a game a few times. We pulled the rules out of the browser so the game could play itself overnight, and it immediately caught a mechanic we had already convinced ourselves was good. That was humbling. It was also the cheapest bug we have ever found.

The problem with testing by playing

Evaluating a game mechanic normally means building it and playing it. This works for questions like “is this fun”. It fails badly for questions like “does this reward skill”. Play a game twenty times with a combo system in it and you will get a feeling about combos. That feeling is a sample of twenty, drawn from your own play, with no control group, and it will be confidently wrong about anything where the signal is weaker than the noise, which in a game whose stones are spawned by weighted random draws describes most of the interesting questions. Ours was.

We hit this with the cascade bonus in Pebble Merge. The mechanic felt good. What we could not tell by playing was whether it fired because we placed a stone well or because stones sometimes happen to line up, and those are very different products. Playing more does not separate them. It only makes you more certain. And the certainty is the dangerous part, because a designer who is sure a mechanic rewards skill will build the rest of the game on top of that assumption, and will not find out it was wrong until much later, if at all.

Splitting the rules out of the browser

The change that made testing possible was architectural, and it was cheap. That surprised me. All the pure rules, meaning the tier ladder, scoring, which tiers can spawn and when the board overflows, live in one file that references neither the DOM nor the physics engine. It is written in a small wrapper that exports the same object whether it is loaded as a script in a browser or required by a test runner in a terminal, and that is the entire mechanism.

That is the whole trick. The moment the rules are a module with no dependencies, a test file can require them and assert on them, and everything else follows. We can check that the eight tunable parameters all exist, that the spawn weights behave, that scoring matches the ladder, and that overflow triggers under the conditions we think it does, all without opening a browser. I keep expecting this to stop being true as the game grows. So far it has not.

Keeping the boundary honest is the ongoing work. The moment a rule needs to know about a DOM element to be evaluated, it stops being testable, and the temptation to reach for one is constant, including for me. We treat any such import in that file as a smell. It is a smell we keep having to remove, and I expect that to continue for as long as anybody is adding features.

Simulating the real engine, not a mock-up

Some questions can be answered with the pure rules. The interesting ones cannot, because they depend on physics, and physics is the part that is easiest to fake badly. Plenty of simulations approximate the thing they are simulating, which is fine for performance estimates and useless for design questions. You end up measuring your approximation, then believing it, then arguing from it, which is worse than having measured nothing and knowing that you know nothing.

So our simulation drives the actual engine, with the actual parameters, and simply declines to render. The physics step runs. The draw call does not. That is the only difference between the game you play and the game the test plays, and keeping it to exactly one difference, rather than to a list of differences you have to keep in your head and re-check every time the renderer changes, is the reason I trust the numbers it produces. Nothing else about the setup is clever.

With that in place, the skill-versus-luck question became a straightforward experiment. Run the simulation under two policies. One drops stones at random. The other aims at the highest stone of the same kind, which is what a moderately competent player actually does. If a mechanic rewards skill, the aiming policy should produce measurably more of it. This is the simplest experiment in the suite, and it is the one I would run first on any new mechanic.

It did not. The time-based combo we liked came out at a ratio of 1.01 between the two policies, which is indistinguishable from luck. The mechanic we almost shipped would have been a reward for throwing stones quickly. We would not have found that by playing, because playing feels like aiming, and I say that as the person who argued it was aiming.

What the suite looks like now

Four files, seventy-two assertions, all running in a couple of seconds on a laptop. Speed matters more than it sounds, and I will come back to why:

The palette suite is the one that surprises people, since it is a test about art. It is there for the same reason as the others. It is a claim we kept making by eye and getting wrong, and eventually we stopped trusting our eyes on it, which is a strange sentence to write about your own colour choices.

A test that exists to say no

The most valuable assertion in the suite is one that encodes a rejected design. When we abandoned the generous cascade multiplier we did not simply delete it. We added a test asserting that the aggressive version inflates score by more than we are willing to accept. The output today reads that the aggressive ladder would have inflated by 66.1 percent, against the roughly 2.4 percent the version we shipped actually costs. Seeing 66.1 in a test log is a better argument than anything I managed to say out loud while we were arguing about it.

This is a guardrail rather than a test. It does not protect against a bug, it protects against us. Six months later, having forgotten the reasoning, somebody, quite possibly one of us, will look at a small cascade bonus and think it should be bigger, and I do not think that somebody is a hypothetical person. The suite will disagree, in writing, with the number attached.

Encoding rejected designs is cheap. I think it is underused. A test that fails when someone reintroduces an idea you already killed is the only durable form of “we decided not to do this”, because the memory of the reasoning lasts about as long as the person who argued for it stays on the project.

What this does not buy you

Being honest about the limits matters more than the technique. A simulation tells you whether a mechanic rewards the behaviour you modelled. It cannot tell you whether a game is fun, whether players will understand a rule, or whether the difficulty curve is right, and it certainly cannot tell you whether the thing was worth building at all, because every one of those depends on humans we are not simulating and on judgements we have not found a way to write down.

Our simulated “aimed” policy models a competent player, not an expert. That is a real limitation and it cuts in a specific direction. It lets us say no simple aiming behaviour predicts deep cascades, and it does not let us say an expert cannot set them up deliberately. We try to state which of those we have shown whenever we quote the numbers, and I would rather over-qualify than have somebody read 1.01 as a claim about experts.

The other thing it does not buy is certainty about feel. Every parameter in our physics tuning was chosen by playing, not by simulating. Simulation killed a bad mechanic. Playing set the numbers. Both were necessary, neither substitutes for the other, and I am wary of anyone who tells you one of the two is sufficient on its own, because in my experience the people most confident about that are the ones who have only ever done one of them.

If you are starting from zero

• Move the rules into a module with no DOM and no engine dependencies. Do this before you need it, because retrofitting it later means unpicking something that has already grown.
• Simulate the real system. An approximation will answer the question you asked about itself, which is a different question from the one you meant.
• Always run a control policy. A number without a comparison is not a measurement.
• Write down rejected designs as failing tests, so the decision outlives the memory.
• Keep the suite fast. Ours runs in seconds, which is why it gets run; a suite that takes ten minutes becomes a suite that runs before releases.

← Back to the blog