How I check AI work before calling it done
A multiplayer village bug, tests I deliberately break, and the human reviews I keep inside a long AI build.
Published 2026-09-02
Loading content...
A multiplayer village bug, tests I deliberately break, and the human reviews I keep inside a long AI build.
I want a coding loop to produce evidence for each completed item, including browser checks when the behavior lives in the browser. In my multiplayer game, watching two clients over ten minutes exposed a village population mismatch a single screen couldn't show.
In a browser game I've been building for a six year old and his friends, the host's village grew over a ten-minute run. The child who joined stayed at exactly 12 villagers.
The host's recorded counts were 12, 14, 17, 19, 21, 22, 22, 23, 24, 24, and 24. Babies appeared and disappeared on both clients, ranging from zero to two. Some villager state was crossing the network. The population wasn't staying in agreement.
Two children could be looking at different villages. Villages were something I'd asked for by name, so this was a missing part of the product even if the first screen looked fine.
The handoff for the next worker contained the counts, the relevant files, and the previous wrong guesses. It pointed to the host's publish path and the joining client's handling of new villagers. It also noted the host's chunk count stayed at 409 during the run.
A fixed chunk count made breeding a more plausible explanation for the growth than new chunk population. It didn't prove the cause. I wanted the implementation agent to trace it and measure, especially because earlier attempts had already gone in the wrong direction.
This distinction matters in a long loop. A hypothesis copied into the next session as a fact can send several more items down the wrong path. I want "observed," "suspected," and "not reproduced" to remain distinguishable in the notes.
I use a task ledger to locate the next item. The evidence in the item document is what makes picking it up useful.
For this game, the soak setup used separate browser instances in one room, with probes for memory, chunks, entity counts, remote players, and frame callbacks. Separate browser instances let the test exercise a real joining client instead of relying on a single page's view of the world.
The room ID determines the world's seed. Changing the room name between a before and after run changes the world as well. If I'm comparing performance, I need equivalent starting world state, the same script, and the same measurement window, or I may be comparing two different workloads.
The prompt also asks for CPU profiling after the join has settled. Starting the profile at launch can measure boot work when the question is about steady-state performance. Production gives the total for the production build; a development run with readable function names helps attribute the work. I keep those results labeled because they're different builds.
Even a simple block check had a trap. In this game, getBlock returns zero for both ordinary air and an unloaded chunk. Reading zero isn't proof I found an empty block in a loaded world. The check has to establish which case it's observing, and it has to inspect the exact cell the test changed.
These are project-specific details. I put them in the handoff because they change whether I can trust the result.
One rule in my implementation prompt is to deliberately break every new test's behavior and confirm the test fails. Temporarily undo the relevant fix, suppress the event it depends on, or otherwise introduce the narrow defect it's supposed to catch. Restore the correct code before running the final checks.
If the test stays green, it isn't checking what I thought it was checking.
I also want a check for over-correction. For the village example, checking only whether additions arrive would miss a fix which inserts the same villager repeatedly. A useful companion check would establish that replaying the addition doesn't create a duplicate. That's an example of the check I'd want, not a claim about a fix verified here.
Sometimes several tests fail for exactly the same deliberate break. I ask for a narrower break which distinguishes their purpose. If there isn't one, there may be a redundant test to remove. I don't need more green lines which all tell me the same thing.
My prompt aims to keep individual automated tests under about a second. The ten-minute soak is a separate measurement. Fast checks make it practical to repeat the loop; a slower investigation is still appropriate when the problem only appears over time.
The prompt says both "never weaken an acceptance check" and "when the browser and your unit test disagree, the browser is right." Those instructions address different failures.
I don't want an agent lowering the requirement to make broken code pass. I do want it to correct a test built on a false model of the runtime, after confirming the browser observation is valid. The product requirement still applies. The record should explain what the old test assumed, what the browser demonstrated, and how the replacement checks the intended behavior.
If I can't automate a check reliably, profiles, screenshots, and documented browser steps can be the evidence. One item in the source handoff, 562, used profiles and screenshots and said so in its document. I would rather see an explicit limitation than a test which cannot detect the defect it claims to cover.
A slow check can also fail because my own test browsers are still consuming resources. The loop needs to stop the processes it started and repeat a contaminated measurement before treating it as a regression. It shouldn't kill unrelated dev servers or make the threshold easier.
The automated checks don't tell me whether a movement feels good, an animation is irritating, or the instructions make sense to the person playing. I expect AI to write most of the documentation for other AI sessions, while I spend more time on these sight, sound, and feel reviews.
I want the checkpoint to be specific enough to act on. Open two clients and watch the same village. Reload a saved world. Try the controls without reading the implementation notes. Then put the feedback into the relevant task document so the next loop can use it.
For the village mismatch, the handoff established a problem and a direction for investigation. It didn't contain a verified after-result. Until the same two-client check passes on the changed build, I wouldn't call that item done.