A benchmark is a snapshot. A track record is evidence.

Consider how differently we treat the two kinds of worker. A person is hired on a history: prior work, references, a probation period, then reviews that continue for as long as they hold the job. An agent is deployed on a demo. It performed well on a set of examples on a Tuesday, so it now runs unattended against live data indefinitely, and nobody looks again unless something breaks loudly.

The gap is not that we trust agents more. It is that we have no instrument for the thing we actually want to know, which is not "is this system capable" but "is this system still doing the work correctly this week."

Point-in-time evaluation tells you a system could. Continuous measurement tells you it does. Only one of those is a basis for delegation.

Why benchmarks stop being informative

A benchmark is a fixed set of problems with known answers. It is genuinely useful for comparing systems at a moment in time, and it is structurally unable to tell you about the moment after. The inputs it tested are not the inputs you will see. It cannot detect a model provider silently changing behavior, a data source drifting, a prompt edit with second-order effects, or a tool whose output format shifted last week.

All four of those are ordinary. None produce an error. The system keeps returning confident, well-formatted answers that are quietly worse than they were, and the benchmark score from launch day says nothing about it because it was measured against a world that no longer exists.

What a track record has to contain

If the goal is evidence rather than a score, a few properties are non-negotiable.

• Repetition on a schedule. One measurement is an anecdote. A series is a trend, and a trend is the only thing that reveals degradation.
• A stable objective. If the task drifts alongside the system, the measurements are not comparable and the whole exercise is theater.
• Separation of failure modes. A rate-limited night is not a quality regression. Conflating infrastructure failure with reliability failure destroys the signal.
• The spread, not just the average. A system that is right ninety percent of the time in two very different ways is two different systems.
• Records the system cannot edit. Self-reported success is not evidence. The measurement has to be written somewhere the measured thing does not control.

The uncomfortable finding

We have been running this measurement on ourselves, and the results have not all been flattering to the multi-agent thesis. Decomposing a document review process and testing each step across several coordination patterns, the honest outcome was that on well-specified structured steps, adding agents added cost and not accuracy. A single capable model found every planted issue. So did four of the five models tested, individually.

The temptation at that point is to keep engineering the task until the swarm wins. That is manufacturing a result. The more useful conclusion was that the value of running several models was never the accuracy: it was that a set of independent models produces a calibrated reading on how settled the answer is. On clear cases they agreed. On genuinely ambiguous ones they split, and that split is a signal a single confident response structurally cannot produce.

Agreement as an instrument

This reframes what measurement is for. It is not primarily a report card. It is the thing that draws the boundary between work that can run unattended and work that needs a person, using data rather than intuition. Where independent systems converge, automate. Where they diverge, route it to a human. The boundary moves as the systems improve, and because it is measured, you can see it move.

How we think about it

We treat measurement as a standing objective rather than a one-off run: defined once, re-measured on a schedule, accumulating into a trend with drift detection when reliability drops in a sustained way. Outcomes are recorded against the agent that produced them, so the performance history attaches to a specific actor rather than to a system in general.

The goal is a straightforward one. Before you hand a real decision to an agent, you should be able to look at what it has actually done, over time, on work like this. Today almost nobody can answer that question. That seems like the thing to fix.