LLM Proposes: Measuring Long-Term Agent Drift

Verifying long-running AI agents presents a unique challenge. When an agent generates its own internal states and reports its progress, and you can't inherently trust those self-statements, you're stuck in a verification loop. A recent arXiv paper by Mohsen Arjmandi, titled "The LLM Proposes, the Executive Disposes," offers a fresh perspective: instead of post-hoc evaluation, embed the verification directly into the system's structure.

This isn't just management philosophy; it's a concrete instrument for agent validation. The core idea is that the Large Language Model (LLM) proposes actions or beliefs, but a separate, deterministic Executive component is responsible for accepting or rejecting them. This shifts the burden of truth from the LLM's internal monologue to a verifiable, external mechanism.

How do you verify a long-running agent when its own states and self-reports are precisely what you can't trust?

From Post-Mortem to Structural Validation

Most approaches to agent verification involve evaluating outcomes after a task is complete, or asking the model to explain its reasoning. The problem with the latter is that the explanation itself is model-generated, essentially using untrustworthy content to validate other untrustworthy content. This paper sidesteps that by having a deterministic Executive control all 'beliefs'. The LLM can only submit typed proposals. For any statement to be accepted as a fact, it must first be a pre-registered prediction made before an action, and then that prediction must be validated against actual observations by code.

In essence, the model loses its ultimate 'memory authority.' It can suggest, it can advise, but whether its suggestions become trusted facts depends entirely on whether code can verify its pre-registered predictions. This mechanism transforms 'self-validation' from a model capability issue into a fundamental structural one.

A Self-Invalidating Experimental Instrument

The paper introduces two particularly interesting features. First, an automatic invalidation mechanism: if certain thresholds are breached—like incorrect shard writes, rendering dimensions, or salted canary echoes—the entire run is marked as invalid. In the initial eight architectural runs, four were invalidated, each successfully pinpointing a real defect. This effectively turns 'experimental failure' into a valuable first-class output.

Second, it employs invisible shadow references. Within each ablation unit, the system compiles the full plan that the complete system would have executed. This shadow reference ensures that even if the mechanism under test is removed, drift metrics remain well-defined, allowing for meaningful comparisons.

Using this instrument, the author reports a clean, single-variable result: ablating the commitment mechanism causes the target abandonment rate to skyrocket from 0.00 to 1.00, while binding error remains stable at 0.00. This was observed across three seeds per grid, with up to 394 reference beats per run, all with valid gating. The binding channel behaved differently; fixing it after ablation didn't cause it to reappear in beat-by-beat drift. This is because binding is held by code, structurally absorbing such failures and only leaving a trace of a hypothesis formation collapse further upstream.

The Value of Honest Zero Performance

Crucially, the paper doesn't shy away from an awkward truth. They ran 52 gated executions on ARC-AGI-3, and the task performance was zero: not a single level was completed. This was pre-registered as a 'structural defeat.' In plain terms, this instrument isn't designed to achieve high scores; it's designed to validate methodologies.

Such candor is rare in AI research. It signals that the authors are more interested in establishing a trustworthy path for agent development than in simply creating a 'better' agent. This is a pragmatic move, focusing on foundational trust rather than immediate benchmarks.

What This Means for Long-Term Agent Developers

If you're building agents with memory, planning, and multi-step execution, this paper addresses a core pain point: distinguishing between 'the model doesn't want to do it' and 'the model can't do it.' The paper breaks these down into commitment drift (abandoning goals) and binding drift (deviating during execution). Measuring these separately is key to knowing which layer needs fixing.

For development teams, several actionable takeaways emerge:

• Entrust critical beliefs to deterministic code, not to the model's self-reports.
• Before acting, write down code-verifiable predictions, then use observations to validate them.
• Treat 'run invalidation' as an integrable meta-signal; it often points to real defects.

However, it's vital to remain clear-eyed: this framework currently shows no task completion ability, as the paper itself admits. It should be viewed as a scientific research tool, not an out-of-the-box agent product. The real challenge with long-term agents isn't single-step accuracy, but understanding where drift begins over extended periods. This paper's value isn't in providing answers, but in offering a structure that allows us to trust those answers.