Testing a level pack you did not write
Unit-testing the game does not tell you whether the 392 levels in this build are solvable. So the suite plays every one of them.
The whole conformance suiteGist · IndieCoreDev/95eb02285852062865e861573d020faaPart 7 of 10 on building a word-block puzzle engine. Part 6: shipping a 4 MB level pack.
The failure mode of a bad content pack is not a crash. It is a level that cannot be solved, discovered by a player, three weeks after release, in a review.
Your unit tests will not find it. They test the code, and the code is correct — it is faithfully rendering an impossible board.
The whole test file
Every game on the engine has exactly one:
import { describeGameShell } from '@gamefactory/word-engine/testing'
describeGameShell(import.meta.url)
That is not a simplification for the article. That is the file.
The suites live in the engine, not in the game, because they check things that must be true of any game built on it. A copy of the suite in each app is a copy that drifts, and the sibling whose copy is stale is precisely the one that ships the broken pack.
It takes import.meta.url rather than a path so a game does not have to know how deep its own
test file sits. The day that changes, it changes in one place.
Four things it checks
// The suite a game runs against its own content pack.
//
// The failure mode of a bad pack is not a crash. It is a level that cannot be
// solved, discovered by a player, three weeks after release, in a review. Unit
// tests will not find it: they test the code, and the code is faithfully
// rendering an impossible board.
//
// So these check things that must be true of ANY game built on the engine, and
// they live in the ENGINE, not in the game. A copy of the suite in each app is
// a copy that drifts, and the sibling whose copy is stale is precisely the one
// that ships the broken pack.
//
// A game's whole test file is therefore:
//
// import { describeGameShell } from '@your-scope/engine/testing'
// describeGameShell(import.meta.url)
//
// import.meta.url rather than a path, so a game does not have to know how deep
// its own test file sits. The day that changes, it changes in one place.
import { describe, it, expect } from 'vitest';
import { appDirOf, readPack, readBrand } from './paths.js';
import { walk, summarise } from './walk.js';
export const describeGameShell = (testFileUrl) => {
const appDir = appDirOf(testFileUrl);
describeShell(appDir);
describePackDeclaration(appDir);
describeOpeningLayout(appDir);
describeBoardWalk(appDir);
};
/**
* The shell still holds nothing but identity.
*
* Structural, and the reason it is worth a test: this is what stops "just this
* once, for this game" from quietly becoming a second codebase.
*/
export const describeShell = (appDir) =>
describe('shell', () => {
it('contains only identity — no components, store or game logic', () => {
const files = listSource(appDir);
expect(files.sort()).toEqual(['src/brand.js', 'src/main.jsx']);
});
});
/**
* brand.json is honest about the pack it points at.
*
* Quiet when wrong: the leaderboard ceiling is stageCount * 3, so an inflated
* count creates a score nobody can reach and a deflated one truncates the
* campaign. Neither throws.
*/
export const describePackDeclaration = (appDir) =>
describe('pack declaration', () => {
it('stageCount matches the catalogue on disk', () => {
const brand = readBrand(appDir);
const { catalog } = readPack(appDir, brand);
expect(brand.content.stageCount).toBe(catalog.stages.length);
});
});
/**
* Every opening arrangement is legible, and is not the answer.
*
* The scramble rule is re-checked HERE, against the shipped data, in a third
* implementation — the solver has one and the runtime fallback dealer has one.
* Three, because it is the constraint whose violation is invisible in a
* screenshot and obvious the moment a player notices it.
*
* A piece's implied board origin is `seat - home`. Two pieces sharing an origin
* are sitting in their solved relationship, which hands the player part of the
* answer. Pieces that are not neighbours in the solution are left alone: at
* their home offset they do not touch, so there is nothing there to read.
*/
export const describeOpeningLayout = (appDir) =>
describe('opening layout', () => {
it('no two filled cells overlap, and nothing overlaps the board', () => {
for (const board of eachBoard(appDir)) {
expect(overlappingCellPairs(board)).toBe(0);
}
});
it('no two solution-neighbours share an implied origin', () => {
for (const board of eachBoard(appDir)) {
for (const [a, b] of neighbourPairs(board)) {
expect(originOf(a)).not.toEqual(originOf(b));
}
}
});
});
/**
* Every board can be walked to a solve — and the shipped metrics are true.
*
* The second assertion is the one worth copying. Three numbers ride on every
* board, written months ago by a different implementation in another repo. The
* runtime walk recomputes them and they must match exactly.
*
* That single check ties together the generator's difficulty model, the ratings
* the campaign order is built from, the power-ups that use the walk, and the
* coin rewards priced off the decision load. If any one drifts from the others
* the game keeps working and starts LYING — prices stop matching difficulty, a
* level rated 8.1 plays like a 4, and nothing throws.
*
* Deliberately not sampled. A sampled content check is a check that passes on
* the run where it mattered: the interesting board is always the one you did
* not draw.
*/
export const describeBoardWalk = (appDir) =>
describe('board walk', () => {
it('every board walks to a complete solve', () => {
for (const board of eachBoard(appDir)) {
const steps = walk(board);
expect(steps.length).toBe(board.pieces.length);
}
});
it('reproduces the metrics the generator shipped', () => {
for (const board of eachBoard(appDir)) {
const m = summarise(walk(board));
expect(m.forced).toBeCloseTo(board.metrics.forcedShare, 9);
expect(m.avg).toBeCloseTo(board.metrics.averageChoices, 9);
expect(m.worst).toBe(board.metrics.hardestStep);
}
});
});
// What this suite cannot tell you, and a green run is persuasive enough that it
// is worth writing down: whether a level is GOOD. Solvable, legible, correctly
// rated and fairly priced are all machine-checkable. Satisfying is not.
Every board can be walked to a solve. The difficulty walk from part 3 is a solver: fill the topmost-leftmost empty cell with any block in hand that fits, and continue. Running it to completion on all 392 boards proves each one has at least one full tiling reachable by the procedure the game itself uses for its power-ups. A board that dead-ends is a board that will strand a player.
Every opening arrangement is legible and not the answer. No two filled cells overlap. Nothing overlaps the board. The clearance gap holds. And the scramble rule from part 5 — no two blocks that are neighbours in the solution may share an implied origin — is re-checked here, in a third implementation, against the shipped data rather than against the solver's intent.
brand.json is honest about the pack it points at. The declaration says stageCount: 392.
The catalogue on disk says how many boards there actually are. If those disagree the build is
wrong in a way that is quiet: the leaderboard ceiling is stageCount × 3, so an inflated count
creates a score nobody can reach, and a deflated one truncates the campaign.
The shell still holds nothing but identity. The structural assertion from part 1. It fails if a component, a store or a piece of game logic appears in an app directory, which is what stops "just this once, for this game" from quietly becoming a second codebase.
The check I value most
Separately from the suite, the economy simulator runs a --test mode that asserts something
narrow and load-bearing: the walk implemented in the engine reproduces, board for board, the
metrics the generator shipped.
Three numbers ride on every board — forcedShare, averageChoices, hardestStep — written by a
tool in another repo, by a different implementation, months ago. The runtime walk recomputes them
and they must match exactly, or the build fails.
That single assertion ties together things that would otherwise drift silently:
- the generator's model of difficulty
- the ratings the campaign order is built from
- the power-ups that use the walk to decide what to place or flag
- the coin rewards priced off the decision load (part 8)
If any one of those four drifts from the others, the game keeps working and starts lying. Prices stop matching difficulty; a level rated 8.1 plays like a 4. Nothing throws. It is exactly the category of bug that is undetectable without a cross-implementation check, and it is why I did not let the runtime walk be an approximation of the offline one.
Testing content is slow, and that is the point
Walking 392 boards is not a fast test. It is seconds, not milliseconds, and it reads a few megabytes off disk.
I let it be slow rather than sampling. A sampled content check is a check that passes on the run where it mattered — the interesting board is always the one you did not draw. If it ever gets slow enough to be a problem, the fix is to run it on the pack-build and the release build rather than on every save, not to look at fewer boards.
Where the suite refuses to help
Two things it cannot tell you, worth being clear about because a green suite is persuasive.
It cannot tell you a level is good. Solvable, legible, correctly rated and fairly priced — all machine-checkable. Whether the board is satisfying is not, and no amount of this replaces playing them.
It cannot check the one board it cannot enumerate. From part 2, a single 13×13 has too many tilings to prove the scoring rule against in reasonable time. It ships flagged. The suite walks it to a solve like any other, which shows it is playable, and says nothing about whether some other arrangement of its pieces scores higher. That is a known, written-down hole rather than an assumption, which is the most I can do with it.
The rule I would generalise
If your content comes from a generator — yours or anyone's — the tests that matter are the ones that re-derive the generator's claims from the shipped artifact, not the ones that check your rendering code.
Every genuine content bug I have had was of the form "the data says X and the data is wrong". A test that mocks the data cannot see any of them.
Next: part 8, pricing an economy off the difficulty model — what happens when reward is a flat rate and the levels are not.