Is my playtest bot cheating? How I tested it

Echo Bat is a one-touch cave flyer. The cave is dark, and the rock only shows when a sonar ping lights it for a moment, or when it is close enough to catch the bat's own faint glow. Tap to flap, don't hit anything.

I wanted a bot to fly it, and the job I had for that bot was specific: tell me whether a person can win this cave. A bot can only answer that if it knows no more than a person does. If it reads where the dark rocks are, it is not playing my game. It is playing an easier game with the lights on, and its score says nothing about mine.

This post is how I checked that it was not cheating, and what I found when it turned out to be honest but still better than a human.

The first autopilot read the answer key

The test build already had an autopilot, added so I could screenshot real gameplay on an Android emulator. It steered by reading the rock positions straight out of the game state. Once the darkness went in, it flew exactly the same with the cave black as with the cave lit. It showed how the game looked. It could not show how hard it played.

The device test runner I use has a built-in solve step, and it was no use here either: it makes one round trip to the device per move. That is fine for a puzzle and hopeless for a game where you tap about twice a second and a late tap kills you. So the bot had to run inside the game's test build, in the frame loop, and send its taps as real pointerdown events through the game's own input.

For a puzzle, a solver that sees the whole board is fair, because the player sees the whole board too. That is how I test a level pack. In Echo Bat the player sees a sliver, so the bot has to be built around that sliver.

One function is allowed to look

I rebuilt the bot with a single rule about structure: exactly one function reads the cave. I call it its eyes. It returns rock only where rock is on screen and lit above a brightness threshold. Moths glow by themselves, so it may see those in the dark. Everything else in the bot works from what the eyes returned:

• memory of what it has seen, each sighting acted on only after a reaction delay;
• feel for how the bat flies, how high a flap lifts it and how fast it falls, which a practised thumb knows;
• no search. It aims at the gap it can see and flaps to hold that height. When the next gap is still dark it holds the middle. It is allowed to lose.

A simplified example of the shape, not the game's code:

// The only place the bot touches the world.
function see(state) {
  const seen = [];
  for (const rock of state.rocks) {
    for (const [x, y] of outlinePoints(rock)) {
      if (!onScreen(state, x, y)) continue;
      if (lightLevel(state, x, y) < MIN_LIGHT) continue;
      seen.push({ x, y });
    }
  }
  return seen;
}

The first test I wrote ran the bot twice per frame, once on the real cave and once on a copy with every rock past the right edge of the screen removed, and failed if the taps ever differed. They never did. That proved the bot ignored the off-screen cave. It proved nothing about dark rock on screen, which is the part that matters in this game.

Asking a second agent to look

I did not trust my own review: I knew what the bot was supposed to do, which is the worst position from which to see what it actually does. So I had Claude Code start a separate agent with a clean context and gave it the requirement, the files, and none of my conclusions.

The brief asked it to trace every piece of information that reaches a tap and put each one in a bucket:

Bucket	Meaning
a	visible on screen right now
b	remembered from an earlier sighting
c	motor feel a practised player has
d	hidden: unlit or off-screen rock, the seed, the rules that place gaps, hitbox forgiveness

Anything in bucket d is a leak. I also listed the subtle ones I could think of: using the full rock list to decide where to look, using exact rock positions as memory keys, a light threshold more generous than what is drawn, a reaction delay applied to some inputs but not others, and any tap that skips the real input path. It was told to change nothing in the repo and to run its experiments in a scratch folder.

Its verdict was that nothing in bucket d reached a tap. The moths were fine because the game draws them whether lit or not. The positions used as memory keys only tracked a seen rock as it scrolled. The bat's own height and speed were read exactly and with no delay, but seeing them 0.1–0.2 seconds late and carrying them forward by feel left the median at 95, so they stayed.

Move what the player cannot see

The classification was a reading of the code. The experiment is what I would trust on its own.

On every frame, the agent took every rock on screen that was fully dark, with all its points below 0.06 light, and moved it 70 units up or down. The moth in its gap stayed where it was. Then it compared what the bot saw and what it tapped against the unmoved cave. It ran 24 caves at phone width, 90 seconds each.

Over 38,416 frames, a moved rock never appeared in what the bot saw. Its sight changed in 190 frames, and every one of those came from the moved rock's shadow falling across rock that was lit. That shadow is drawn on screen, so a player sees it too: a fair cue, not a leak. Its taps differed in 5 frames, all downstream of those shadows.

That is the test worth copying. If moving something the player cannot see changes what the bot does, the bot is reading it. I turned it into a permanent test. In the full suite it first took 7.9 seconds, over the 5-second limit per test, so it now flies 20 seconds of cave with a 30-second timeout.

It has a limit, and the shadows were the first hint of it. The test only works when the hidden thing does not change what is visible. Later the cave gained winding tunnels whose walls are the obstacle, and walls block the sonar wave, so moving a dark wall changes the light elsewhere on screen, and the test would flag a difference a player can see. For walls the bot got two other tests instead: every wall point its eyes return must be on screen and lit above the threshold, and a scan of the bot's own source fails if any cave-reading function appears outside the eyes.

It did not cheat. It was just too good

The audit's second finding was the more useful one. The bot was honest about the cave and still better than a person, in ways that moved the score a lot. The agent swept each setting and measured the median score:

Setting	Values tried → median score
Light needed to see rock	0.10 → 95, 0.15 → 69, 0.20 → 48, 0.25 → 39, 0.30 → 16
Reaction time	0.10 s → 95, 0.15 s → 93, 0.20 s → 56, 0.25 s → 41
Touch lag, fixed 50 ms, not planned for	→ 48
Remembered rock height off by ±8	→ 97
Bat height misjudged by ±8	→ 104

The last two rows mattered as much as the first three. They say perfect memory and perfect judgement of height were not where the bot's edge came from, so there was no point blurring them. The light threshold, the reaction time and the touch lag were.

The agent's case on reaction time was that 0.1 seconds is the floor for a bare reflex, and picking a gap is a decision. The light threshold I did not want to guess, so I measured it. In emulator screenshots taken in flight, the background sat at 2–3 out of 255, and rock rims next to the bat reached 243–247. That matched how the game draws a rim: about 247 times its light. A threshold of 0.1 meant a rim around 25 on near-black, which the agent judged visible on an OLED screen in a dark room and not in daylight. At 0.16 the rim is about 40, which is about where the faint outlines in those screenshots were still visible.

What changed:

	Before	After
Light needed to see rock	0.1	0.16
Reaction time	0.1 s	0.15 s
Touch lag	none	50 ms ± 20 ms, and the bot aims for where the bat will be when the tap lands
Fastest repeat tap	0.07 s	0.12 s
Where it taps	the game canvas directly	whatever is under the cave's centre

The last row is the small one. The bot was dispatching its taps straight to the canvas, so if the settings popup was open over the cave, it would still flap the paused bat. A real finger hits the popup. Tapping whatever element sits at the cave's centre makes a popup take the tap, as it would on a phone.

With all of it on, the median over 40 caves at phone width went from 95 to 45, and 32 of the 40 runs still beat 25, the score the device test needs. The agent's own harsher mix of limits had given 36; the difference is in what was included, and I did not split it further.

What a fair bot is worth

An honest bot changes what its deaths mean. A crash by a bot that sees everything only tells you about its flying. When this one crashes into rock it never saw, the cave hid that rock too long for a player with those limits, and that is a finding about the game. The playability test in the suite now counts those deaths separately and fails if they pass 3%.

It also reacts to things a cheating bot would ignore. Before the limits, a 400-unit-wide screen gave a median of 142 against 95 at phone width. More screen, more warning, better score, which is what a player would get too.

The leak that came later

Later the same day the camera started following the bat up and down, and the eyes were updated so rock counted as on screen only inside the new vertical window. Moths were still checked left to right only. Reviewing that change, I found the bot could see moths above and below the screen, and since every moth sits in a gap, that meant finding gaps a player could not. The fix was one condition. The test that pins it moves the screen a full height up, then down, and expects no moths at all.

None of the existing tests looked above or below the screen. "On screen" is a definition, and it changes whenever the camera does.

Where it stands

Before the limits, the bot passed the device scenario on an emulator in 43 seconds: start with a best of 25, fly past it, stop, and expect "New best". After the limits I could not run it on the device the same day: half-finished cave changes were sitting in the same folder, and the app built from them did not start. The headless numbers above are from the simulation, not from a device.

If you want to try this on your own game:

• Give the bot one function that reads the world, and test that nothing else does.
• Move what the player cannot see and assert that nothing changes. Then check whether hiding something changes what is drawn, because then this test cannot work.
• Have someone, or something, that did not write the bot classify its inputs.
• Treat "too good" as its own audit. Mine never read a hidden rock, and human limits still halved its score.

It is the same habit as breaking a check to watch it fail: a test for "it uses nothing hidden" proves nothing until you have hidden something and watched.