Skip to content
team banzai

tech demo · software

Predicting which files a fix will touch
before it's written

If I touch this, what else will need touching?

The question is not only a programmer's: in any tangled system, a tariff that feeds others, a template contract, a chained spreadsheet, touching one thing drags along others you don't see until they break.

a report comes in. where will the fix land?

Three real reports from the test, exactly as they came in. The system does not read the code: only the title and body of the report, and the project's history frozen at its cutoff. Press one and watch where it points.

its five candidates, in order

  1. 1 the grouping core matches the real fix
  2. 2 the grouping methods
  3. 3 the grouping operations
  4. 4 the shared core of tables and series
  5. 5 the tables

A hit: the first candidate matches the real fix, and the rest of the top are its co-change neighbors.

The territory: 15 pandas files with their role in place of their name. Each line joins two files that changed together (thickness: how many times, before the 2023 cutoff); each node's size, how often it was touched. In purple, the system's five candidates for each case, with their co-change ties; in amber, where the real fix landed. Real cases from the test, untouched.
The real files behind each role

The three reports are public pandas issues: numbers 45231 and 50634 (hits) and 45263 (the miss; its fix also touched the type declaration of the same null-value module). The roles on the map correspond to these files:

  • the group definition · pandas/core/groupby/grouper.py
  • the grouping core · pandas/core/groupby/groupby.py
  • categorical grouping · pandas/core/groupby/categorical.py
  • the grouping methods · pandas/core/groupby/generic.py
  • the grouping operations · pandas/core/groupby/ops.py
  • the tables · pandas/core/frame.py
  • the columns · pandas/core/series.py
  • the shared core of tables and series · pandas/core/generic.py
  • the indexes · pandas/core/indexes/base.py
  • formatting and printing · pandas/io/formats/format.py
  • the version report · pandas/util/_print_versions.py
  • the optional dependencies · pandas/compat/_optional.py
  • the package setup · setup.py
  • the low-level null-value code · pandas/_libs/missing.pyx
  • the low-level date code · pandas/_libs/tslib.pyx

summary

the data
The pandas code repository on GitHub: fifteen years of public history, with over 25,000 resolved problems and 37,000 merged changes.
the question
When someone reports a new bug, can you guess which files will need work, before anyone fixes it?
the test
The system only knows the project's history up to a date. For each bug reported afterward, it proposes its candidates. Then we open the real fix and compare.
what came out
In one out of four problems, the system's very first candidate was the right file. Looking at its five candidates, the right one was there in more than half of the cases. For comparison: always betting on the most-edited files gets the first pick right less than one time in ten.

It is the question everyone who maintains a large system asks. To answer it we used the full history of pandas, one of the most widely used software libraries in the world: fifteen years of changes, with every problem and every fix documented in public.

The experiment: when someone reports a problem, can a system that only knows the project's history predict which files the fix will touch, before it exists? Without reading the code closely: only the text of the problem, which files changed together over the years, and which words live in which corners.

The usual rules. We froze the history at four cutoffs (2021 to 2024). The system only sees what came before each cutoff; the test problems arrived afterward. The parameters were frozen before we looked at a single result, with a cryptographic fingerprint (769c68f). And a decision that cost us almost half the test: we discarded every problem whose text was edited after it was created, because we cannot guarantee that the version we see is the original. We preferred a smaller, clean test over a large, riggable one.

What came out

Across 1,122 future problems: the system placed a file from the real fix in its first position 28% of the time, and among its five candidates 58%. Always betting on the historically most-touched files stops at 8% and 25%. And the memory of "what changes together" adds on top of text alone: without it, first position drops to 22%.

the right file, on the first try

the system 28%
text only 22%
co-change only 15%
betting on the usual files 8%
How often the first candidate is already the right file. Among the top five, the system reaches 58% (betting on the usual files, 25%).

Three cases, with the future open

They are the three on the map above, exactly as they came out of the test. A hit: someone reports that grouping an empty table throws an error. The system proposes as its first candidate the central file of the grouping subsystem, exactly where the real fix landed, and fills the rest of the top with files from the same neighborhood.

Another hit: grouping by categories, on an empty table, fails. The first candidate was the file that defines how groups are formed, exactly where the real fix lived.

And a miss: a report about "null values printing wrong" dragged the system toward printing and environment files, when the fix lived in the low-level null-value code, which barely shares vocabulary with the report. Lexicon alone is not always enough; that case counts as a failure.

How to read this without fooling yourself

  • No triage labels. The labels on an issue are added by a maintainer AFTER diagnosing, using them would be putting the answer into the question. The system sees only the original title and body.
  • The test is about localized fixes (one problem, one fix, few files). The fraction that survives that filter is published; a giant refactor cannot be predicted this way and we do not pretend otherwise.
  • Ceiling declared: 3.5% of the fixes touched only files that did not exist when the history was frozen. Impossible by construction; they count as failures.
  • Numbers with intervals. Each percentage carries its 95% confidence interval.

The caveats, hiding none

  • A single project. pandas is large and modular; in a project with a different structure the number will be different. Validation on a second project is proposed as a continuation.
  • 44% of the candidate problems were excluded due to later text edits (not verifiable). The full cut is published, case by case.
  • It is a test on the closed history of a public project, not a universally validated tool.

References

This idea has twenty years of literature behind it, and anyone who says otherwise is selling you snake oil. What this demo adds is the verifiable protocol: history frozen per cutoff, uncontaminated input, pre-registered parameters, and the ceiling and the failures published.

  • Zimmermann, Weißgerber, Diehl & Zeller (ICSE 2004), Mining Version Histories to Guide Software Changes (the ROSE tool; extended version in IEEE TSE 2005).
  • Zhou, Zhang & Lo (ICSE 2012), Where should the bugs be fixed? (BugLocator).

Data and acknowledgements

This one stands on the work of the pandas project and its contributors, out in the open for fifteen years now. Thank you.

No affiliation with the project. The examples cite public files and problems, never the people.

Can you use this? Yes. Our analysis, the map and the texts on this page are published under the CC BY 4.0 licence: use them, share them or analyse them freely, citing the source, Team Banzai (team-banzai.com). Third-party data (the public history of the pandas repository on GitHub) keeps the terms of its original sources, and the project's code keeps its own licence.

Built with Python and SQLite. Lexical retrieval over paths and commit messages + a co-change graph, blended with frozen weights. No language model decides anything.

About this demo

Retrospective study for illustrative purposes, on the public pandas repository. All cited problems are historical and resolved. This analysis does not evaluate the project or its contributors, and is not a technical recommendation about the analyzed software.

What about your company?

The same method, applied to a factory's fault history, tells you which parts tend to be replaced together; applied to a contract archive, which clauses are always touched at once; applied to a company's processes, which departments get dragged along when you change one.

If one of these experiments reminds you of a problem of yours: a repo only two people understand, changes that break files nobody linked together, teams touching the same thing without knowing it, write to us and we'll talk it through. No fluff: we'll tell you whether it can be done.

← All tech demos