LongRCA Bench: Diagnose Long-Agent Failures

When an AI agent fails after dozens or hundreds of actions, the final error is usually the least useful part of the story. A task may go off course much earlier, during a handoff, an incorrect assumption, or a decision that looked reasonable at the time. LongRCA Bench is designed to test whether an AI system can find that turning point rather than merely describe the bad outcome. The benchmark comes from an arXiv preprint focused on failure diagnosis for long-running agent trajectories, where debugging requires reconstructing a process instead of checking a single answer.

That distinction matters in practical systems. A short chatbot exchange can often be reviewed by reading a few messages. A multi-agent workflow is different: several roles may contribute, context can be passed between them, and a mistake may remain hidden until much later. LongRCA Bench asks a model to inspect the full execution trace, identify the earliest decisive root-cause step, and determine which role was responsible. Those are related questions, but they are not interchangeable.

A benchmark built around natural failures

The dataset contains 1,140 failed trajectories spanning five domains. These are not examples where researchers inserted an obvious mistake at a known location. The failures occurred naturally during task execution, making the diagnostic problem closer to what an engineering team encounters in production. Each trajectory includes expert annotations for both the responsible role and the earliest root-cause step, with annotation consistency evaluated separately.

The median trajectory length is 145 steps, which is a meaningful challenge for current reasoning systems. In a long trace, the visible failure may be separated from its cause by many successful-looking actions. A model that focuses only on the last few events can produce a plausible explanation while still missing the point at which recovery was no longer possible. The benchmark’s structure is intended to expose that weakness.

• 1,140 naturally occurring failure traces across five domains
• Median trajectory length of 145 execution steps
• Expert labels for both responsible role and earliest root cause

For developers, the useful idea is not simply “use longer logs.” It is to preserve the sequence and ownership information that makes those logs diagnosable. A trace that records only the final response cannot answer whether a planner, tool-using agent, reviewer, or handoff introduced the problem. LongRCA Bench turns that missing context into an explicit evaluation target.

Why ordinary reasoning baselines struggle

The strongest reported baseline achieves only 13.2% exact root-cause accuracy. That result is revealing because it shows how difficult long-horizon attribution remains even when a model can read the complete trajectory. Broad explanations are easier than precise localization. A system may correctly recognize that a plan failed or that a handoff was problematic, yet still identify the wrong step as the earliest decisive cause.

The paper introduces Root-Cause Trajectory Attribution (RCTA), a training-free method aimed at making that search more manageable. Its workflow uses summaries of trajectory segments to retrieve candidate error steps, then traces the evidence backward toward earlier handoff instructions. This is a pragmatic design: instead of asking a model to reason over every event with equal attention, it narrows the search and uses the structure of the workflow to revisit likely causes.

RCTA raises exact root-cause-step accuracy to 24.1% and reaches 51.1% responsible-role accuracy. The improvement over the strongest baseline is substantial, but the absolute scores also temper any claim that long-agent debugging is solved. A role can sometimes be identified without finding the exact step, while a candidate step can look suspicious without proving that it was the decisive cause. Reporting both numbers makes that gap visible.

What agent teams can take from it

LongRCA Bench gives observability a sharper definition. In a complex agent application, a successful final answer is not enough, and a failed final answer is not a diagnosis. Teams need process evidence: which role acted, what context it received, what instruction it handed off, what tools were called, and when the trajectory first diverged from the task objective. Without those fields, automated analysis is forced to guess from incomplete records.

A useful starting point for an internal system is to make every execution event carry a stable step identifier and role label. Handoff messages should be retained alongside the surrounding context rather than flattened into a final transcript. Developers can then test whether a diagnostic tool can separate two tasks that are often confused:

• Attributing responsibility to the role most associated with the failure.
• Locating the earliest step that made the eventual failure likely or unavoidable.

That separation is more than a reporting preference. A workflow owner may need role-level accountability to decide where to add safeguards, while an engineer debugging a prompt or tool interface needs the exact transition that caused the drift. Evaluating only one target can make a system appear stronger than it is. LongRCA Bench’s design encourages teams to keep those measurements independent.

What to watch as the benchmark develops

The benchmark is especially relevant to researchers building agent evaluators, tracing platforms, and automated incident-review tools. It offers a common setting for comparing methods on failures that are long, naturally generated, and annotated by people rather than manufactured around a known error. The current results also leave plenty of room for improvement. A 24.1% exact-localization score indicates that models still struggle to connect late symptoms with the earliest meaningful decision.

For anyone testing an agent today, the practical lesson is straightforward: preserve richer traces before trying to build a root-cause system. Store role ownership, handoff instructions, relevant context, tool outcomes, and step order. Then evaluate role attribution separately from exact-step localization. LongRCA Bench does not remove the difficulty of debugging long-running agents, but it provides a more honest way to measure it—and a useful reminder that the path to a failure often matters as much as the failure itself.