tech demo · software · legacy
Rebuilding a software's documentation by reading its code, exam included
We gave a system the source code of DOOM, changed four numbers behind its back and asked it to document how it works. It caught all four: it reported the changed value, with the exact line as proof, while the documentation humans have written about DOOM for thirty years still says the old one.
And so you don't have to take our word for it, the game runs right here, in your browser, compiled from that same changed copy. You open a door and a stopwatch measures how long it waits before closing: 3 seconds, the ones the docs claimed, not the usual 4.3. The proof stops being read and starts running.
Pick a system your company depends on: the ERP installed in 1998, the program that runs payroll, the one that decides which orders go through. It works. But the person who wrote it left years ago, the manual in a drawer describes a version from three overhauls ago, and nobody dares touch it because nobody fully knows what it does inside. The knowledge isn't lost: it's written, line by line, in the code. Only nobody reads it.
The capability we test here is exactly that: rebuilding how a system works from the real code, not from its manuals, and proving it isn't making it up. We show it with DOOM because its code is public, under a free licence, and anyone can check what we claim. The case is DOOM; the pattern works for your ERP.
And since documenting isn't one single thing, we generate four documentations from the same reading of the code, each in the format a different team uses: the technical one for whoever will maintain it, the functional one for the business analyst, an agile backlog of user stories for a Scrum team, and the plain manual for whoever just uses it. All four are right below the game, and you can open and download them.
The same code, running: time the gotcha
We don't show the code on one side and the game on the other: the engine below is compiled from the code we documented. Start it, open a door with the spacebar and watch the stopwatch top right. It isn't painted: it measures the engine's own clock (35 steps per second) between the door finishing its opening and starting to close. Switch between the usual copy and ours with the button, and you'll see the count change.
0.00s
build: canonical (the usual one)
GPL engine · webassembly
DOOM, from our copy
The engine runs in your browser, compiled from the code with the four changes. It only downloads when you hit start (about 30 MB); nothing heavy loads before that.
The four documentations, one per team
This is what the system produced, and it is the point of the demo: the same reading of the code, told four ways, each in the format a different team already uses. The technical one for whoever maintains the code, the functional one for business and QA, the agile backlog for a Scrum team, and the manual for whoever just plays. That is the thesis: the same knowledge, in the format your team uses.
Every claim with a number carries its file and line: click any and you open id Software's public code at that exact line. You'll see the usual value (150, not 105), because GitHub has the public code and here we document our copy with the four numbers changed. That difference is exactly the experiment, and further down we tell you how we set it up.
The door cycle, generated by reading the code in front of it. Every number carries its line and its confidence grade; no value is claimed that isn't on a line that was read.
Time and speed constants
VDOORSPEED = FRACUNIT*3 p_spec.h:364.
With FRACUNIT = 65536, that's 3 map units per tic for the normal door.
· high confidence
VDOORWAIT = 105 tics p_spec.h:365.
At 35 tics per second doomdef.h:122,
that's 105/35 = 3.00 seconds of wait at the top.
· high confidence
Method note: VDOORWAIT and VDOORSPEED aren't checked against the Doom Wiki (which may reflect another build); they're checked by reading p_spec.h in this tree and, if wanted, by timing the door in the binary compiled from this code. That's exactly what the stopwatch above does.
State structure: vldoor_t
Defined at p_spec.h:343-360.
The direction field is 1 (rising), 0 (waiting at top) or -1 (lowering); on
reaching the top, a normal door goes to direction = 0 and starts the counter
topcountdown = topwait p_doors.c:182.
State machine: T_VerticalDoor
The "thinker" that runs once per tic p_doors.c:63
branches by direction: rising, it moves the plane toward the ceiling
p_doors.c:168;
waiting, it decrements the counter and, on reaching zero, orders the close and plays the
close sound p_doors.c:81;
lowering, if something crosses it, it reopens instead of crushing
p_doors.c:151.
Constants outside the door (part of the exam)
Bullet damage: damage = 7*(P_Random()%3+1) p_pspr.c:633,
i.e. 7, 14 or 21 per hit. Clip:
clipammo[am_clip] = 15 p_inter.c:59,
a clip gives 15 bullets (cap 200).
Safe change points
| I want to change… | I touch… | Blast radius |
|---|---|---|
| Door wait at the top | VDOORWAIT p_spec.h:365 | every door with topwait = VDOORWAIT |
| Door speed | VDOORSPEED p_spec.h:364 | normal door; the fast ones use VDOORSPEED*4 and scale too |
| Bullet damage | multiplier at p_pspr.c:633 | pistol and chaingun, both via P_GunShot |
| Bullets per clip | clipammo[0] p_inter.c:59 | clip pickups and starting ammo |
Declared uncertainty
How far a door rises in absolute units depends on the level geometry, not on a constant: you can't claim a fixed height without a specific map. Only the wait at the top (VDOORWAIT) is a fixed, quotable time in seconds.
Observable rules, not code. For an analyst, QA or product: what can be seen and timed, with its evidence and how to check it.
R2 · The door waits 3.00 seconds at the top before closing
The wait is VDOORWAIT = 105 tics p_spec.h:365; 105 ÷ 35 = 3.00 s. How to check: time it from fully open until it starts lowering; it should be ~3.0 s, not ~4.3 s. · high confidence
R3 · The normal door moves at 3 units per tic
Speed VDOORSPEED = FRACUNIT*3 p_spec.h:364, consumed by the movement engine p_doors.c:171. How to check: measure the rise tics over a known distance. · high confidence
R8 · A pistol bullet does 7, 14 or 21 damage
Hit damage is 7*(P_Random()%3+1) p_pspr.c:633: it falls in {7, 14, 21}, not {5, 10, 15}. How to check: instrument the shot or seed the generator and log the damage applied. · high confidence
R9 · A clip gives 15 bullets
clipammo[am_clip] = 15 p_inter.c:59; the bullet cap is 200 p_inter.c:58. How to check: pick up a clip with the counter visible and watch the +15. · high confidence
Other rules read
A locked door demands the right-coloured key and grunts if you don't have it p_doors.c:376; a door that catches you while lowering reopens instead of crushing you p_doors.c:151; an enemy can open normal doors but not close them p_doors.c:430.
Declared uncertainty
The total open time in seconds isn't a fixed rule: it depends on the gap height (level geometry) divided by the speed. Damage is random within a set: the correct claim is "one of 7, 14 or 21", not "a fixed value".
The same knowledge as a Scrum backlog: three epics, twelve user stories and acceptance criteria in Gherkin (Given / When / Then), ready to automate as tests. Priority uses MoSCoW; the point estimate is illustrative (it measures relative effort a team agrees on with its own velocity, and there is no such team here). What is verifiable are the citations: every criterion with a number carries its file and line, just like the other three.
The three epics
A · A door's life cycle: it opens when used, rises, waits up top for a fixed time, comes down on its own and doesn't crush whoever crosses it p_doors.c:168. B · Access control and door variants: coloured-key locks, fast doors, timed doors and the three ways to open one. C · Hitscan weapons and ammo: how much a bullet takes off, how much a shot spends and how much a clip gives.
Sample stories (12 in the PDF)
US-02 · The door closes on its own after the wait. As a player I want the door to close on its own so I don't have to close it by hand.
US-10 · Each bullet does variable damage. As a player I want each bullet to do damage so I can kill enemies, with some variation per shot.
US-11 · A clip refills ammo up to the cap. As a player I want to pick up clips to keep firing, without going over the maximum.
What this backlog does not cover
The total opening time of a door in seconds isn't an estimable story without a concrete level: it depends on the WAD's geometry. And bullet damage is random within a set: the correct criterion is "one of 7, 14 or 21", not a fixed number per shot. Automating it means seeding the random generator.
Plain, no jargon. The numbers come from the real code; the reference in brackets is there in case anyone wants to check it, but you can ignore it if you just want to play.
Doors
Many DOOM walls are actually doors. You walk up and press the use key (spacebar). A normal door rises until open, stays open about 3 seconds and closes on its own (105 tic wait, p_spec.h:365; at 35 steps per second, 3.0 s). If it catches you underneath while closing, it doesn't crush you: it reopens (p_doors.c:151).
Some coloured doors need a card or skull key of the same colour. Without it they won't open and you hear an annoyed grunt. And some fast doors slam open and shut at quadruple speed: don't dawdle underneath.
Your pistol
Each shot spends one bullet. Each bullet that hits does between 7 and 21 points of damage, varying each shot (7, 14 or 21, p_pspr.c:633). When you pick up a clip you gain 15 bullets (p_inter.c:59), and you can stock up to 200 in total.
In one line
Open doors with the spacebar (some need a key), don't stand underneath when they close, and remember each clip gives you 15 bullets and each bullet does 7 to 21 damage.
How we do it: we change the exam without telling it
There's a trap to deal with before we start, and we say it out loud: DOOM is probably the most dissected program in history. There are whole books about it, and any modern AI knows it by heart. So asking it to "document DOOM" proves nothing: it recites it back. The good question isn't whether it knows DOOM, it's whether it documents the code in front of it or recites what the Doom Wiki says.
To find out, we changed four numbers in the code behind its back: how long a door waits before closing, how fast it moves, how much damage a bullet does and how many bullets a clip holds. Four numbers, changed in place, without telling it. Then we asked it to document how all of that works.
It's the teacher's trick of changing a number in the question to see who read and who copied from the book. Say the new number, you read the code. Say the book's number, you copied. And the nice part is that it checks itself: every claim in the doc carries its file and line, and the marker goes to that line to see whether the number is really there.
The result, with the trick in plain sight: world, change, docs
Like a magician's audience that does see the trick: for each number, three columns. What the world says (the Doom Wiki and id Software's public code), what we changed in secret, and what the generated docs ended up saying. Your eye jumps on its own: the docs match what we changed, not what the world says.
The real code, with the line lit up
You don't need to know C. Click a claim and it jumps to id Software's real file with the line that proves it lit up in amber. It's the same public code the citations link to: you'll see the usual value there (150, not 105), because GitHub has the public copy and we documented our changed one. That difference is exactly the experiment.
int topwait;
// (keep in case a door going down is reset)
// when it reaches 0, start going down
int topcountdown;
} vldoor_t;
#define VDOORSPEED FRACUNIT*2
#define VDOORWAIT 150
void
EV_VerticalDoor
( line_t* line,
mobj_t* thing ); case 1:
// UP
res = T_MovePlane(door->sector,
door->speed,
door->topheight,
false,1,door->direction);
if (res == pastdest)
{
switch(door->type)
{
case blazeRaise:
case normal:
door->direction = 0; // wait at top
door->topcountdown = door->topwait;
break;void
P_GunShot
( mobj_t* mo,
boolean accurate )
{
angle_t angle;
int damage;
damage = 5*(P_Random ()%3+1);
angle = mo->angle;
if (!accurate)
angle += (P_Random()-P_Random())<<18;
P_LineAttack (mo, angle, MISSILERANGE, bulletslope, damage);
}
// a weapon is found with two clip loads,
// a big item has five clip loads
int maxammo[NUMAMMO] = {200, 50, 300, 50};
int clipammo[NUMAMMO] = {10, 4, 20, 1};
//Highlighted with Shiki at build time, over the real C file. The line numbering matches the GitHub link.
4/4
gotchas caught: it reported the changed value, not the usual one, in all four
87/87
traceable citations: all 87 file:line references across the three docs point to a real line
0/4
negative control: docs that recite the usual values score zero, the instrument knows how to fail them
2/3
docs with confidence grades and an uncertainty section (the user manual leaves it out on purpose)
The negative control is what separates a test from a trick: we fed the marker a set of docs that recite the usual values, and it failed them on all four. If the instrument couldn't give a zero, passing would mean nothing.
Take it: the method, not our word
What we've done here you can do yourself with any documentation written by an AI. We leave you the tools: the plain-language checklist, the marker we used to score, and the four sets of documentation already typeset, each in its field's standard format.
The checklist for auditing AI docs
Nine common-sense checks, in plain language, that turn "trust me" into "I'll look myself":
- Every claim cites its source (no citation, doesn't count).
- The source really says that (you go and check).
- A change made behind its back gets caught (the exam-question trick).
- The marker knows how to fail (if it always passes, it's worthless).
- Declares what it doesn't know · separates fixed from variable · separates fact from inference · speaks the reader's language · is reproducible.
The marker, open
The same corrige.py that
produces this page's tally: it goes to each cited line, checks the number by hand and decides
with no judgement of its own. We publish it with the mutations and the reproducible report, so
you can run it. It's heuristic, not a semantic parser, and we say so: its two false negatives
are in "the caveats".
repo link as soon as it's published
The caveats
What this first version doesn't prove yet, said plainly.
And in your company?
Swap DOOM for what actually keeps you up at night: the ERP nobody dares migrate, the payroll program only the person who left understood, the business rules buried in a procedure nobody has reread. The knowledge is still there, written in the code. This pulls it out, tells it for whoever maintains it, whoever decides and whoever uses it, and leaves the exact line as proof it wasn't made up.
If this sounds like a problem of yours, get in touch and we'll talk it over. No fuss: we'll tell you whether it can be done or not.
What it's built with, no inflation. The documentation was written by a language model reading the code: that's the part that drafts and connects. But what decides pass or fail isn't the model: it's a deterministic marker in Python that goes to each cited line and checks the number by hand, with no judgement of its own. The model proposes; the marker, which has no opinion, verifies. The four mutations are deterministic patches over the code, reproducible byte for byte, and the 87 citations resolve against the real file. The game is id Software's GPL engine compiled to WebAssembly with Emscripten; the stopwatch reads the engine's own clock (our patch, published with the fork), it doesn't simulate it.
Licences and attribution
This demo exists because id Software released the DOOM source. Thank you.
The DOOM engine is id Software's, released under the GPL-2.0 licence. The code we show and the code we compile to WebAssembly come from github.com/id-Software/DOOM, commit a77dfb9, tree linuxdoom-1.10, via the doomgeneric port. Our copy with the four mutations and the stopwatch patch is distributed under GPL-2.0 too, as that licence requires. The game assets are Freedoom Phase 1 (BSD licence). DOOM is a trademark of id Software; this is not endorsed by id Software or ZeniMax.
Can you use this? Our text and analysis on this page are published under a CC BY 4.0 licence: use them citing the source, Team Banzai (team-banzai.com). The DOOM code keeps its GPL-2.0 licence.