At a look
- Drawback: Debugging AI agent failures is tough as a result of trajectories are lengthy, stochastic, and infrequently multi-agent, so the true root trigger will get buried.
- Answer: AgentRx (opens in new tab) pinpoints the first unrecoverable (“important failure”) step by synthesizing guarded, executable constraints from instrument schemas and area insurance policies, then logging evidence-backed violations step-by-step.
- Benchmark + taxonomy: We launch AgentRx Benchmark (opens in new tab) with 115 manually annotated failed trajectories throughout τ-bench, Flashand Magentic-Oneplus a grounded 9-category failure taxonomy.
- Outcomes + launch: AgentRx improves failure localization (+23.6%) and root-cause attribution (+22.9%) over prompting baselines, and we’re open-sourcing the framework and dataset.
As AI brokers transition from easy chatbots to autonomous programs able to managing cloud incidents, navigating advanced internet interfaces, and executing multi-step API workflows, a brand new problem has emerged: transparency.
When a human makes a mistake, we are able to often hint the logic. However when an AI agent fails, maybe by hallucinating a instrument output or deviating from a safety coverage ten steps right into a fifty-step activity, figuring out precisely the place and why issues went unsuitable is an arduous, handbook course of.
At this time, we’re excited to announce the open-source launch of AgentRx (opens in new tab)an automatic, domain-agnostic framework designed to pinpoint the “important failure step” in agent trajectories. Alongside the framework, we’re releasing the AgentRx Benchmark (opens in new tab)a dataset of 115 manually annotated failed trajectories to assist the group construct extra clear, resilient agentic programs.
The problem: Why AI brokers are onerous to debug
Fashionable AI brokers are sometimes:
- Lengthy-horizon: They carry out dozens of actions over prolonged durations.
- Probabilistic: The identical enter may result in totally different outputs, making copy tough.
- Multi-agent: Failures might be “handed” between brokers, masking the unique root trigger.
Conventional success metrics (like “Did the duty end?”) don’t inform us sufficient. To construct protected brokers, we have to establish the precise second a trajectory turns into unrecoverable and seize proof for what went unsuitable at that step.
Introducing AgentRx: An automatic diagnostic “prescription”
AgentRx (quick for “Agent Prognosis”) treats agent execution like a system hint that wants validation. As a substitute of counting on a single LLM to “guess” the error, AgentRx makes use of a structured, multi-stage pipeline:
- Trajectory normalization: Heterogeneous logs from totally different domains are transformed into a standard intermediate illustration.
- Constraint synthesis: The framework robotically generates executable constraints based mostly on instrument schemas (e.g., “The API should return a legitimate JSON response”) and area insurance policies (e.g., “Don’t delete information with out person affirmation”).
- Guarded analysis: AgentRx evaluates constraints step-by-step, checking every constraint solely when its guard situation applies, and produces an auditable validation log of evidence-backed violations.
- LLM-based judging: Lastly, an LLM decide makes use of the validation log and a grounded failure taxonomy to establish the Vital Failure Step—the primary unrecoverable error.
A New Benchmark for Agent Failures
To judge AgentRx, we developed a manually annotated benchmark consisting of 115 failed trajectories throughout three advanced domains:
- τ-bench: Structured API workflows for retail and repair duties.
- Flash: Actual-world incident administration and system troubleshooting.
- Magentic-One: Open-ended internet and file duties utilizing a generalist multi-agent system.
Utilizing a grounded-theory strategy, we derived a 9-category failure taxonomy that generalizes throughout these domains. This taxonomy helps builders distinguish between a “Plan Adherence Failure” (the place the agent ignored its personal steps) and an “Invention of New Info” (hallucination).
| Taxonomy Class | Description |
|---|---|
| Plan Adherence Failure | Ignored required steps / did additional unplanned actions |
| Invention of New Info | Altered info not grounded in hint/instrument output |
| Invalid Invocation | Instrument name malformed / lacking args / schema-invalid |
| Misinterpretation of Instrument Output | Learn instrument output incorrectly; acted on unsuitable assumptions |
| Intent–Plan Misalignment | Misinterpret person aim/constraints and deliberate wrongly |
| Underneath-specified Person Intent | Couldn’t proceed as a result of required information wasn’t out there |
| Intent Not Supported | No out there instrument can do what’s being requested |
| Guardrails Triggered | Execution blocked by security/entry restrictions |
| System Failure | Connectivity/instrument endpoint failures |
Key Outcomes
In our experiments, AgentRx demonstrated vital enhancements over present LLM-based prompting baselines:
- +23.6% absolute enchancment in failure localization accuracy.
- +22.9% enchancment in root-cause attribution.
By offering the “why” behind a failure by way of an auditable log, AgentRx permits builders to maneuver past trial-and-error prompting and towards systematic agentic engineering.
We consider that agent reliability is a prerequisite for real-world deployment. To assist this, we’re open sourcing the AgentRx framework and the entire annotated benchmark.
We invite researchers and builders to make use of AgentRx to diagnose their very own agentic workflows and contribute to the rising library of failure constraints. Collectively, we are able to construct AI brokers that aren’t simply highly effective, however auditable, and dependable.
Acknowledgements
We wish to thank Avaljot Singh and Suman Nath for contributing to this mission.
