sealed exam · public data
Matching two lists of the same money,
without the key that links them
Reconciling means matching two lists that talk about the same money but don't reference each other properly: the payments that left the account on one side, and the contracts or obligations that justify them on the other. In a company it's the bank statement against the general ledger; in a public administration, the payments executed against the contracts awarded. When the reference matches, it's routine. When it's missing or badly written, someone spends the afternoon chasing what each entry corresponds to. Anyone who has squared an account at month end knows that work.
In the other demos we seal a prediction before knowing the future and wait for the future to prove us right or wrong. Here the data already exists: the payment and its contract come linked from the factory by an official key. So we change the trap. Instead of hiding the future from it, we hide the key: we conceal the contract identifier that already links them and ask the engine to rebuild the link without it, using only what would remain in a real case, the amount, the dates and the supplier's name. The hidden key is a public, unquestionable identifier, so the exam is criticism-proof: anyone can download the same data and repeat it.
The verdict, up front: by the criterion we sealed before looking at anything, the engine does not win. It shines where the payment equals the year's obligation, with 96.2% accuracy, and falls short on partial payments, which were three out of every four. The whole story, with its numbers, is below.
Each dot in the sea is one of the exam's 12,649 payments, in its real colour. The larger, ringed dots are concrete cases: click them to see why the engine matched or failed that payment.
The exam's result, in figures
Out of 12,649 matchable payments: the amount-and-date heuristic decides 1,871 with 99.2% accuracy; the language model decides 3,262 with 26.6% accuracy; the engine abstains on 7,515 payments, 59.4%. Exact reference matching decides 0 by construction, because the key is hidden.
The verdict, and why this is not a win
Before running the engine over the corpus, we sealed the success condition and anchored it with a timestamp. Winning meant two things at once: recovering more payments than three simple baseline rules (more coverage) and making fewer mistakes than the best of them (precision no worse). The engine meets the first by a hair, recovering 21.5% against the best simple rule's 20.7%, and fails the second: it gets 53.1% of what it decides right, against the 100% scored by the dumbest rule of all (if a supplier has a single contract, the payment belongs to it). Under any reasonable reading of "the best rule", the engine's precision falls short. A draw or a defeat gets published under that name, and this is a defeat.
That it comes out this way is what sustains the credibility: it's the kind of result that only appears when the exam is sealed before running it and not retouched afterwards. The good part is measured too, and it is emphatic. The cause fits in one sentence: the exam lived in the partial payments, and there the work falls on a small local language model that gets one in four right.
Match by match, which layer ties each one
The engine is not a black box: it's a cascade of layers tried in order, from the safest to the riskiest. First the exact reference match; then amount and date; then the language model over the contract description. Hiding the key switches the first layer off completely, and that's not a bug, it's the exam's design: the reference tying payment and contract is exactly what we concealed. The whole weight falls on two layers, and they split the work very unevenly.
Zero by construction: the reference is hidden, so the layer that ties by it cannot weigh in. It's what makes the exam easy to understand.
When the payment equals the obligation, matching by amount and date almost never fails. It's the job the engine is built for, and it does it with the official key hidden.
Where only a fraction of a multi-year contract is paid, the amount stops anchoring and the only layer that can weigh in is the local language model, 7 billion parameters and zero cost. It gets one in four right: this is where the engine gives ground, and it's the margin the next instalment measures.
How to read the bars: each one is that layer's precision, how many of its decisions were correct. The exact reference bar sits at zero on purpose. The memory of previous reconciliations and the sum of partials also score zero here, the first because there is no history of this case in public data, the second because the instalments that would add up to the total are paid in years that aren't in the corpus.
The four sealed variants
The exam ran four times, crossing two axes: the supplier's name cleaned or exactly as typed, and its identifier hidden or visible. The case we show is the realistic hard one, no identifier and a clean name. Abstention is the percentage of payments where the engine says "I don't know" instead of forcing a pair.
| Variant | Precision | Coverage | Abstention |
|---|---|---|---|
| No identifier, clean name (the case we show) | 53.1% | 21.5% | 59.4% |
| No identifier, raw name | 52.6% | 21.4% | 59.4% |
| With identifier, clean name | 54.2% | 21.6% | 60.1% |
| With identifier, raw name | 53.1% | 21.5% | 59.5% |
All four fit within little more than a point and a half of precision. That they barely move is itself a result: neither cleaning the name nor showing the identifier moves the needle, because the heavy lifting is done by the amount, which doesn't look at the name, and the bottleneck isn't telling similar suppliers apart but telling apart the many contracts of one same supplier.
The three regimes, separately
Mixing the three hides that some are much easier than others. The pre-registration asked for them to be separated, and here is why it mattered: the exam lived in the middle row.
| Regime | Payments | Decides | Precision | Coverage |
|---|---|---|---|---|
| The payment matches the obligation 1:1 | 2,783 | 2,443 | 96.2% | 84.4% |
| Partial and multi-year | 9,739 | 2,667 | 14.0% | 3.8% |
| Negative corrections | 127 | 24 | 4.2% | 0.8% |
Partials are 9,739 of the 12,649 matchable payments, nearly 77%. There the amount stops anchoring the payment to its contract, the weight shifts to the text, and that's where the engine gives ground. If the corpus were only payments that match 1:1, this report would be a comfortable win: the engine ties them with 96.2% precision and recovers 84.4%, with the key hidden.
Four failures, each with its story
These aren't cases picked to look good: they are the archetypes the pre-registration anticipated, pulled from the results files with their real numbers. They share one root: amount proximity is a treacherous clue when a supplier concentrates hundreds of contracts with overlapping figures.
The supplier collision
payment P000000077 · Caltech / Jet Propulsion Laboratory
The institute that operates NASA's propulsion laboratory has 1,762 candidate contracts, almost all with the same space research label. The payment disbursed $3,105,898; the engine picked, with confidence 0.95, a contract whose obligation was $3,114,736, a near carbon copy of the amount. The true one was another, worth $14.2 million. It chose by amount proximity among nearly two thousand contracts identical in appearance, and got it wrong. In this supplier alone it makes 392 mistakes.
The mistied partial
payment P000000242 · Lead Builders Inc
A partial payment of $48,447, barely 1% of a large contract. It had only 14 candidates and still failed: it proposed a $4.02 million construction contract ("reduce arc flash risk") when the true one was another worth $4.20 million ("chiller replacement"). Since the payment is a tiny fraction, the amount can't tell which, and the language model had to decide by the description and picked the wrong one.
The correction filed where it doesn't belong
payment P000000658 · Amentum Services
A one-dollar refund, negative amount. An adjustment like that carries no amount signal to tie anything with, and the engine should have abstained; instead it forced an answer with confidence 0.9 and chose wrong. The negative corrections regime is the exam's worst: 127 payments, a single hit. A naive rule of matching to the nearest positive amount wouldn't even know where to put them.
The giant-block multi-year partial
payment P000000080 · Jet Propulsion Laboratory
The payment was $1,051,548 and the engine proposed, with confidence 0.95, a contract with an obligation of $1,039,900, again a near copy of the amount; the true one was worth $7.2 million. The failure's structure is the same as the supplier collision's: when a supplier concentrates hundreds of multi-year contracts and the payment is a fraction, the amount points at the wrong contract and the language model has nothing to correct it with.
The caveats, and the exam that audited itself
This does not recreate two independent books, bank against ledger: it hides the key of a system that already came linked from the factory. The advantage is that the hidden key is unquestionable, an official identifier, and that makes the exam repeatable by anyone. The limit is that reconciling two genuinely independent books, with free-text names and misaligned dates, is the next phase. We also declare that to consider multi-year payments the date window had to be widened, a profile fixed in advance by the nature of the data, not an adjustment made while looking at the result.
The order came first: the pre-registration was frozen in a commit with a timestamp proof anchored in the Bitcoin chain, before running the engine over the full corpus. Afterwards two inaccuracies surfaced and were corrected in public, with their reason and their scope, not covered up. One: the pipeline claimed "zero contract identifiers in the descriptions" and two remained stuck to other letters, one of them a leak citing somebody else's contract; the cleaning was reinforced and only the text field the language model reads changed. The other: the mechanism passing the pairs to the model sometimes left the payment unevaluated, so some abstentions weren't prudence but a hole in the transport; it was fixed and verified payment by payment. Credibility doesn't come from never being wrong, it comes from correcting yourself in full view.
What's next
The model that can handle the partials. This exam deliberately used a small local model, zero cost, to set the floor. Almost all the room for improvement lives in that language layer, and the next instalment runs the same exam, with the same corpus and the same key hidden, changing only the model. The x-ray by levels allows a clean comparison: you'll see exactly how much it climbs.
Where names do rule. This corpus arrives linked by an official key; the hard reconciliation is between two independent books that only share the supplier's name in free text. That is the terrain where a name resolution layer has to earn its keep, and it's the phase this exam leaves ready to measure.
And in your company?
This very thing, with your bank statement and your general ledger: matching every receipt and every payment to the invoice, contract or entry that justifies it, and flagging the ones that don't square so someone looks at them. In a company's books the entry carries rich text and a contract's instalments fall within the year, which are exactly the two things this exam lacks, so the engine has more to work with, not less.
We used United States data for a concrete reason: Spanish administrations don't publish both sides at once, the payments executed and the contracts that justify them, with a key linking them. NASA does, through the public USAspending portal, and that lets us hide the key and measure without asking anyone for data. If your accounts have entries squared by hand every month, reconciliations that always depend on the same person, or a close that slips because someone is chasing what each receipt corresponds to, write to us and we'll talk it over. No fuss: we'll tell you whether it can be done or not.
Data and technical notes
This experiment exists because there are institutions that publish their data in the open. Thank you.
- Corpus. NASA payments from fiscal year 2024 against its contract catalogue from 2019 to 2024: 12,649 matchable payments against 41,091 candidate contracts. Public data from USAspending, downloaded and verified on 16 July 2026.
- Seal. Pre-registration frozen in a commit with a timestamp proof anchored in Bitcoin. First the seal, then the engine over the full corpus, with nothing recalculated.
- Model. A 7-billion-parameter language model running locally, temperature zero, zero cost, with no data leaving the machine.
- Frozen configuration. Date and amount tolerance, confidence threshold and window fixed beforehand by the multi-year nature of the data, not tuned while looking at the result.
No affiliation with NASA or with USAspending. The reconciliation errors are ours alone.