tech demo · software
Flagging which changes deserve a second look before merging
A second look, before it hurts.
It applies to more than code: any flow where almost everything is routine and every so often something deserves a second pair of eyes, a payment outside the usual range, an account created with too many permissions, a config change made live.
this change deserves a second look
merging a small change, in a file that had been dormant for over a year
It is not that it is guilty: it is that waking up old code is exactly what, in this project's history, ends up undone most often. It deserves an extra pair of eyes.
the cost, always beside the benefit
2.1×
more reverts in the flagged band than in the average
73:1
changes braked for every revert caught
That second number, not the first, is the business decision: a reasonable toll if the second look is cheap; hugely expensive if not.
summary
- the data
- The VS Code and Kubernetes repositories on GitHub: 300,000 real code changes, with the full record of which ones had to be undone.
- the question
- Every so often a change goes wrong and has to be undone. Can you know in advance which changes deserve an extra review?
- the test
- Looking only at what was known at the moment of each change, the system flags a few as "look twice". Then we check which ones actually ended up undone.
- what came out
- In the band the system flagged there were twice as many undone changes as the average. And the surprise: the same detector, taken to another project, stopped working.
The earlier demos looked at the archive to understand what already happened. This one attempts something more uncomfortable: to have the archive grab your arm BEFORE something goes wrong.
The setting: two of the most active software projects in the world (the VS Code editor and Kubernetes). Hundreds of changes merging every week. Every so often one goes wrong and has to be reverted: undone, officially and on the record. The question: looking only at what was known at the moment of each change, can you flag which ones have an abnormal probability of ending up reverted?
Important: this is NOT about finding culprits. Identities do not even enter the analysis. And it claims no causes: it is a precautionary brake, like a doctor ordering one more test because of your history. It points to where to look twice, nothing more.
The usual rules, plus one extra measure. Each change is evaluated only with the history up to that instant. Parameters frozen with a cryptographic fingerprint (42ad6d7). And the system trains on one project and is tested, without touching a single parameter, on another completely different one.
Run it through the brake: four real changes
Four changes from the VS Code test stretch (2021-2024), scored with the frozen parameters. No author, no identifier: only what the brake saw at that moment. Pick one, watch the signal, then check the real outcome.
the exam, live 42ad6d7
deserves a second look
probability of ending up undone 3.1%, against the test average of 0.7%. It crosses the frozen threshold: flag.
what the brake saw
- The file had gone 4.3 years untouched. Waking up dormant code is the strongest signal in this project.
- The person signing it had 294 changes in the project, but none in that file.
the toll Of every 73 changes the brake stopped in the test, only 1 actually ended up reverted. The flag pays off if the second look is cheap.
the outcome, from the real record
flag was right Reverted 57 days later. One of the 126 the brake caught.
What came out
In VS Code (46,333 test changes), the band the brake flagged concentrated 2.1 times more reverts than the average, capturing 41% of all of them. The confidence interval excludes chance, and the advantage over the best simple detector is clean: the intervals do not even touch.
the lift, with the humility check
73:1 changes braked for every revert caught
The humility check, also published: the same brake, transferred to Kubernetes without touching a parameter, barely distinguishes itself from chance (1.3 times, with an interval that grazes 1). There is no universal brake: every house has its own cracks, and you have to learn them from its own history.
Two things we didn't expect
- Big changes are NOT the dangerous ones. Change size has a negative correlation with ending up reverted. The "careful, that's a big change" instinct does not survive the backtest.
- The fragile thing is the dormant thing. The strongest signal: files that had gone a long time untouched. Waking up old code is what calls for a second look.
How to read this without fooling yourself
- We measure recorded reverts, not bugs. A bad change that was patched without a revert does not show up. The brake predicts "this will end up undone", which is the public trace of the problem, not the whole problem.
- It works without the easy tell too. Removing the signal "this file was already reverted before" (which a skeptic would rightly call tautological), the result holds: 2.15. The merit was not the trick.
- The threshold was frozen before the test and in the test it braked 19.7% of the changes, not the 10% expected: the scores drifted. No moving the bar after the jump.
- Work-in-progress reverts excluded: 60% of VS Code's reverts are by the author themselves on the same day (undoing a half-finished step). Counting them would have inflated everything; the filter is robust (relaxing it to 2-3 days moves nothing).
The caveats, hiding none
- Detection bias: a revert exists because someone saw it and decided to revert. The negatives contain silent failures that nobody undid.
- The "went wrong" label comes from the history itself: it is the recorded reverts, identified automatically, with no case-by-case human review. Some of those reverts are product decisions, not failures, and we have not separated that contamination by hand.
- 8.7% of the reverts could not be traced to their original change, and that group is biased toward large changes: it is declared and bounded.
- It is a study of two public projects with their specific review culture; the numbers do not travel on their own (the transfer to Kubernetes proves it).
References
Per-change risk prediction is a field with its classics, and the giants use it internally. What this demo adds: public pre-registration, calibrated probabilities, the cost of the policy published, and a transfer to another project that came out weak and is reported.
- Kamei et al. (IEEE TSE 2013), A Large-Scale Empirical Study of Just-in-Time Quality Assurance.
- Industrial per-change risk systems at Meta (Diff Risk Score) and Google (speculative testing).
Data and acknowledgements
We could build it because VS Code and Kubernetes develop in the open, with their full history in view. Thank you.
No affiliation with Microsoft or the CNCF. Identities hashed before the analysis; no public example allows anyone to point to a person.
Can you use this? Yes. Our analysis, the backtest probabilities and the texts on this page are published under the CC BY 4.0 licence: use them, share them or analyse them freely, citing the source, Team Banzai (team-banzai.com). Third-party data (the public histories of VS Code and Kubernetes on GitHub) keeps the terms of its original sources, and each project's code keeps its own licence.
Built with Python and SQLite over the git history. Logistic regression with isotonic calibration, features computed at the instant of each change. No language model decides anything.
About this demo
Retrospective study for illustrative purposes, on the public repositories of VS Code and Kubernetes. It measures recorded reverts in historical changes already resolved; it does not measure software quality nor evaluate the projects or their contributors. Author identities are kept out of the analysis and of every visible surface.
What about your company?
The same idea works for flagging which accounting entries deserve a second look before you close the month, for pointing out which purchase orders fall outside the norm before they are approved, or for warning which new customer accounts are worth checking twice before you sign them off.
If one of these experiments reminds you of a problem of yours: deploys that scare you without telling you why, a thousand-line change and a three-line one reviewed the same way, checks that run on gut feeling instead of where it hurts, write to us and we'll talk it through. No fluff: we'll tell you whether it can be done.