Advising on a never-seen ticket
with years of past incidents
The memory you already paid for.
Any team that has spent years resolving incidents has that archive; the rare part is
actually using it when the next never-seen case comes in.
the query, live
Two real tickets from the blind test, exactly as they came out. Pick one and watch what
the system did. Ticket titles and quotes are in their original English; the system
answers in Spanish, the language of the run.
query
answers with citations
comes inFLINK-35848 · [Release-1.20] Vote on the release candidate
answers (excerpt, in the Spanish of the run)
Este ticket es una tarea estándar del proceso de release de Apache Flink (release 1.20): iniciar y gestionar el hilo de votación [VOTE] en la lista [email protected] para el release candidate. No es un incidente técnico, sino un paso operativo del release. Los análogos recuperados (…) son exactamente la misma tarea de "Vote on the release candidate" y muestran cómo se resolvieron.Este ticket es una tarea estándar del proceso de release de Apache Flink (release 1.20): iniciar y gestionar el hilo de votación [VOTE] en la lista [email protected]para el release candidate. No es un incidente técnico, sino un paso operativo del release. Los análogos recuperados (…) son exactamente la misma tarea de "Vote on the release candidate" y muestran cómo se resolvieron.
"The vote will be open for at least 72 hours. It is adopted by majority approval, with a…"
The real resolution matches in substance. Both automated judges, blind to each other: full hit.
abstains
comes inFLINK-35408 · Add 30 min tolerance value when validating the time-zone setting
answers (in the Spanish of the run)
No encuentro en el histórico casos suficientemente parecidos a este ticket. ¿Puedes indicar la versión exacta de Flink, el componente afectado y el mensaje de error completo (stacktrace) para poder ayudar con más precisión?No encuentro en el histórico casos suficientemente parecidos a este ticket. ¿Puedes indicar la versión exacta de Flink, el componente afectado y el mensaje de error completo (stacktrace) para poder ayudar con más precisión?
the reason
No similar cases and not enough evidence. Rather than inventing a weak proposal, it asks for context.
Both judges agreed: correct abstention. Abstaining without evidence is also getting it right. It is the differential, not a defect.
The advice always comes with the cases that back it; you can go to the source and
check. Abstaining, when there is no basis, is also getting it right.
summary
the data
The ticketing system (Jira) of the public Apache Flink project: the same kind of ticketing app any company uses, with nearly twelve thousand resolved, documented issues.
the question
Every company keeps years of tickets: problems that came in and how they were solved. Can that memory give advice when a new problem arrives?
the test
The system only knows the old tickets. We show it new tickets it has never seen, without their solution, and ask it for advice. Then we compare against the real solution.
what came out
The system's advice was useful in 8 out of 10 graded cases. And when it had no basis, the system said "I don't know", which is also getting it right.
Every company hoards the same archive: the ticket queue. Years of problems,
discussions and solutions, written down, stored, and never read again. For this
demo we used a public one: a large software project with nearly twelve thousand
resolved, documented issues.
The question: when a new problem walks in, can a system use that memory to give
advice? And we don't mean finding similar words. We mean advice: what may be going
on, how this got solved before, and with how much confidence. With the right to
abstain: we prefer a system that says "I have no basis for an opinion" over one that
always has one.
We built the memory only from tickets earlier than a cutoff date.
Later tickets were shown to it without their resolution. We wrote the scoring
criteria before looking at a single result, and every answer was graded by two
independent automated judges, from different providers, who never see each other's
reasoning; when they disagreed, the worse grade stood.
What came out
Across the future tickets that could be graded with the list of criteria written in
advance (the "rubric"), the advice was useful (full or partial hit) in
79.5% of cases. It abstained correctly in 9.6%.
It failed in 11%. And on one in six tickets it reviewed, it said "I have no basis
for an opinion", abstention is part of the design, not a defect.
the funnel
tickets resolved in the test period934
with a resolution reconstructible from the archive523
sampled for evaluation120
gradeable with the rubric73
the verdict over the 73 gradeable ones
32.9% full hit 46.6% partial hit 9.6% abstained for good reason 11.0% miss
Useful advice (full + partial): 79.5%. Criteria written before looking. Abstention is part of the design.
The detail we like most: the advice always comes with the historical cases that back
it. It is not an oracle. It is the project's memory, organized and cited; you can go
to the source and check.
How to read this without fooling yourself
The funnel is published whole. Of 934 tickets resolved in the test period, 523 had a resolution reconstructible from the archive itself, and within the graded sample, 47 were resolved with code alone, no written explanation, those cannot be graded with the rubric.
Yes, there are language models working here. That is the experiment: advice in natural language, with a safety net, citations verifiable against the source, dual independent automated judges, and mandatory abstention when evidence is missing.
"Useful" has a written definition. Full or partial per a five-category rubric frozen before grading; the two automated judges agree in 86% of cases.
Abstaining is getting it right when there is no evidence. Advice without a basis is not advice: it is noise with good syntax.
The caveats, hiding none
The automated judge has not been checked against systematic human grading; the numbers are those of the pre-registered blind test, with the rubric frozen in advance.
Ticket texts come from the system's final dump, not a snapshot frozen at resolution time: a later edit could leak information. A declared limit, common to every living archive.
This is a software development project, not a support desk: 39-44% of cases are resolved with code alone. In a real support archive that fraction changes, and the demo would need repeating there.
It is a rubric-graded sample, not a production deployment. It shows the memory is usable, not that any particular product works.
References
Recommending solutions from historical tickets is a field with its own literature
(case-based reasoning, ticket recommendation, retrieval augmentation). What this
demo adds is the protocol in public: the future hidden, criteria frozen before
looking, abstention measured as a first-class result, and the full funnel published.
Data and acknowledgements
There is a demo here because the Apache Software Foundation works in the open and its
issue archive is public. Thank you.
No affiliation with the foundation or the analyzed project. Ticket authors appear
pseudonymized on every visible surface.
Can you use this? Yes. Our analysis, the backtest results and the
texts on this page are published under the
CC BY 4.0
licence: use them, share them or analyse them freely, citing the source,
Team Banzai (team-banzai.com). Third-party data (the public issue
archive of the Apache Software Foundation) keeps the terms of its original sources.
Built with TypeScript and PostgreSQL. Semantic retrieval over the archive and language
models to draft the advice, with dual independent automated judges from different
providers and mandatory abstention.
About this demo
Retrospective study for illustrative purposes, on the public issue archive of an
Apache Software Foundation project (no affiliation). All cited tickets are historical
and resolved. This analysis does not evaluate the project or its contributors, and is
not a technical recommendation about the analyzed software.
What about your company?
It works just as well for giving a new hire the answer the veteran would give to an
objection, for helping a technician diagnose a breakdown from years of past
interventions, or for guiding whoever handles a query that has already been solved a
hundred times, just never by that person.
If one of these experiments reminds you of a problem of yours: a support queue where every answer gets reinvented, the good fix buried in old tickets, what's already been solved a hundred times getting solved from scratch again,
write to us
and we'll talk it through. No fluff: we'll tell you whether it can be done.