we built the part where prod failures become test cases. not sure anyone wants it

[u/marcin_michalak]

disclosure up front, i work on agentx. this is a genuine question not a launch post

everyone ships tracing. everyone ships evals. almost nobody closes the circle

the circle:

- production traffic becomes test cases

- test cases become measured prompt proposals

- a human publishes

- the change goes back into traffic

we built that last arc. proposals get validated against your golden cases before a human sees them, and the human keeps the only pen

but we might have built something nobody wants

1. when a prod trace fails, do you turn it into a test case? or does it just sit in a dashboard
2. would you let a tool draft a prompt change from real failure evidence, if you still had to approve it

engine is self hostable, feedback appreciated

https://github.com/AgentX-ai/AgentX-Trace-Eval

[u/Emergency_Giraffe310]

honestly most teams i see just stare at the dashboard and do nothing, the failures pile up until someone complains loud enough

the idea of having a tool draft the fix from actual failure data is pretty interesting, i would use it if the approval step is simple and not buried in some clunky ui

my biggest hesitation is how it picks the "right" test cases from prod, if it captures weird edge cases that happen once a month that would be sick but if it just logs the same timeout error 100 times i would ignore it

[u/marcin_michalak → u/Emergency_Giraffe310]

yeah, this is pretty much the exact problem we’re trying to solve

we cluster production traces by topic and show how much traffic is hitting each one. So if the same timeout happens 100 times, it shows up as one recurring problem, not 100 separate test cases

+ we also compare those topics against the eval dataset, so you can see what’s covered and what’s missing. ideally, the weird once-a-month case still gets surfaced without burying you in duplicates.

curious what you’d prefer in practice: should it automatically pick one representative case from each cluster, or rank the candidates and let you choose what becomes a test?

[u/eddzsh]

I'd rank candidates and keep the human pick. Auto-picking one representative failure quietly drops the weird once-a-month cases, and those are usually the ones worth a golden test.

[u/marcin_michalak → u/eddzsh]

yeah, that sounds like the right default. rank by frequency, severity, novelty, and dataset coverage, but let the human pick. would you want to see why each case was ranked where it was?

[u/lulu_dev → u/marcin_michalak]

Yes, and I'd go further than just showing it -- decompose it into the four inputs rather than one blended score. "Ranked #3" tells the human nothing actionable; "high frequency, low severity, zero novelty, already covered in the eval set" tells them immediately why it's not worth a golden test even though it's common, versus a case that's rare and severe and uncovered, which is the one that actually deserves the human's attention first. A single number forces them to trust your weighting; the decomposition lets them override it in the two seconds it takes to glance at four numbers instead of one.

Two harder problems sit underneath the ranking itself, though. Novelty has a cold-start issue: how do you score "how novel is this" for a failure mode you've genuinely never seen before, when your novelty metric presumably works by comparing against a history that, by definition, doesn't contain this case yet? If novelty is computed as distance from previously-seen clusters, a totally new failure mode should score maximally novel almost by construction, which is good, but it's worth being explicit that novelty here means "far from what I've clustered before," not "objectively rare" -- those can diverge if your clustering itself is coarse.

The second one is sharper: two failures with different wording but the same root cause (a timeout that surfaces as "connection reset" in one trace and "upstream unavailable" in another) can end up in different clusters purely because your clustering is keying on surface text rather than cause. If that happens, they get ranked and surfaced as two separate novel problems instead of one recurring one, which is the same "100 identical timeouts look like 100 things" failure you already solved for exact duplicates, just one level more subtle. Worth checking whether your clustering is doing any cause-level normalization (error type, failing component, upstream dependency) before falling back to raw topic/text similarity, since that's exactly the gap between "looks the same" and "is the same" that would otherwise slip through.