Four Agents, One Plan

Correction (September 2026): This post was rewritten against the benchmark's own records. The earlier version put work that an agent session did into my voice (the first results, the pooling defect, the miscounted cells), and it opened on a framing that the session proposed and put to me as a question, with no answer on record. It also counted four vendors where there were three and five refreezes where there were four. It said no retries where availability failures could be retried off the ledger, called the synthesis attenuations within noise with no test computed, and credited the jail with part of Grok's low score although that share is unidentified. It also said that all three implementation defects threatened the headline result, when the grid defect touched only the secondary decomposition, and it misstated which of Grok's packets hit the wall clock and how far the other workers' synthesized scores moved. The companion report it linked is not yet published.

This benchmark was preregistered: its plan, tools, answer key and inferential test were frozen and committed by hash before any worker ran, and its results went through two rounds of review by a separate model before publication. It asked a deliberately narrow question. Take a plan, a decomposition of a research task into concrete work units, written once by a single strong model, and hand that identical plan to four coding agents as workers. Which agent executes the plan best against a hidden answer key, and when a synthesis model pools each agent's output separately, does the pooling preserve the differences between the agents or wash them out? The design proposal, reviewed twice by GPT-5.6 Pro, labels this estimand narrowed and states explicitly that the pilot does not claim the planning model is a better orchestrator than the alternatives. On July 9, 2026 I resolved the decisions the proposal left to me, choosing maximum effort for the GPT-5.6 lanes and Sonnet 5 as a worker, and approved this track as a pilot. An agent session then built the apparatus, ran it and drafted the first version of this write-up.

One plan, four workers

The plan was fourteen task decompositions authored in advance by Claude Fable 5 and frozen behind a commitment hash before any worker ran. Each decomposition splits a research task into work units the size of a packet. A worker reads a packet, searches a fixed document corpus through a metered file drop channel, and emits structured evidence records (an entity, a criterion, a source document and a verbatim excerpt) that are later scored against a gold key it never sees. Fable's role ends at the plan, since it is the orchestrator and not one of the workers. The four workers are the contestants: GPT-5.6 Sol driving the codex CLI at maximum effort, Claude Opus 4.8, Claude Sonnet 5 and Grok 4.5, which makes four agents from three vendors, with the two Claude models sharing one. The matrix held 276 scored runs, with one retained binding realization per manifest row and no reruns chosen by score, although availability failures could be retried off the ledger.

Every worker ran inside its own isolation, so that nothing but the model and its harness varied. The two Claude workers ran under a filesystem jail with a tool allowlist, and the codex worker ran in its own kernel sandbox, which permits writes only inside its workspace. Grok was the one worker whose web channel could not otherwise be closed at the harness layer, so it ran inside a rootless kernel network jail. The jail gave it its own network namespace with the default route deleted, an nftables egress firewall, and a proxy passing only allowlisted hostnames as the only path out. The gold key, the manifest and the scoring criteria were absent from every jail.

The apparatus writes to a ledger that can only be appended to, each entry chained to the one before it by its hash. The ledger's head hash was posted to a timestamped channel before any number was computed, so the outcome was committed before it could be tuned to a result. Every published figure below is recomputed in a witnessed mode that checks the ledger on disk against that posted hash and refuses to run if the two disagree.

What the scores said

Scored on the primary decomposition across twelve tasks, the mean F1 ranks the four workers cleanly: GPT-5.6 at 0.663, Opus 4.8 at 0.641, Sonnet 5 at 0.529 and Grok 4.5 at 0.295. The preregistered statistic is the mean difference in F1 per task for each ordered pair of workers, and on that statistic the point estimates tell the same ordered story. GPT-5.6 leads Grok by 0.368, Opus leads Grok by 0.346 and Sonnet leads Grok by 0.234, while GPT-5.6 leads Sonnet by 0.134, Opus leads Sonnet by 0.112 and GPT-5.6 leads Opus by only 0.022. These are point estimates for twelve tasks, and the preregistered test certifies far less of them than the ranking suggests.

What the preregistered test resolved

The preregistration fixed the inferential test in advance, and the confidence intervals are not that test. The frozen test is a permutation test with Holm adjustment across the family of all six pairwise contrasts, while the bootstrap intervals are, by the same preregistration, a diagnostic of how sensitive a result is to which tasks happen to be in the suite. The first draft of these results reported four of the six gaps as significant, because their marginal 95 percent intervals excluded zero. That reading ignored the frozen familywise test, and testing six contrasts one at a time lets the multiplicity that a family test corrects for pass as significance. Under the Holm correction only one gap survives. GPT-5.6 beats Grok at an adjusted p of 0.047, just under the line, and every other contrast is unresolved at twelve tasks, including the three whose marginal intervals looked clean.

The one thing this benchmark established, then, is that the strongest worker beats the weakest. The fuller ordering, with GPT-5.6 and Opus ahead of Sonnet and Sonnet ahead of Grok, is the point estimate, and the results record calls it suggestive, not established: it would need more tasks to resolve. The review caught the overclaim before publication.

Whether synthesis preserved the differences

The second question was the distinctive one, although the preregistration labels worker execution its primary output and synthesis its secondary. After the workers run, a synthesis model (Opus, at the CLI's default sampling) takes the evidence one worker produced for a cell and pools it into a single synthesized answer. That answer is scored against the same gold key with the same metric, and each worker's output is pooled separately. The quantity of interest is attenuation: whether the gap between two workers shrinks when their synthesized outputs are compared instead of their raw ones. If synthesis rescued a weak worker or collapsed a strong one, the result would be a large attenuation.

On this run the ranking was unchanged, and every observed attenuation was at most 0.045 in magnitude; no test of attenuation was computed. The synthesized means per worker (0.662, 0.637, 0.484 and 0.295) track the raw ones closely. The single visible movement is that Sonnet 5's synthesized output scores about 0.045 below its own worker output, which nudges every pair that includes it, while the other three workers moved by at most 0.004 and Grok's synthesized and raw scores are identical. The analysis bootstraps the two gaps that make up each attenuation separately but never resamples their difference. The uncertainty of that difference depends on a covariance between the paired outputs that was never computed, so each attenuation is a descriptive point estimate and not a tested effect. Described at that strength, synthesis on this run neither rescued the weak worker nor collapsed the strong ones.

What Grok's number measures

Grok's 0.295 is not a clean measurement of extraction quality. Of its twelve primary cells, three were zeroed because every packet in them hit the 600 second wall clock while Grok ran at maximum effort inside the network jail. Four more of its cells completed inside the budget and still scored zero, because the evidence Grok returned credited nothing in the gold key, and those are genuine extraction misses rather than timeouts. On the remaining five it scored between 0.47 and a perfect 1.0. The 0.295 therefore blends a model that did the task well on five cells and sometimes finished with nothing the key could credit, together with three cells lost to the wall clock under the jailed condition. No control was run without the jail or with a longer packet wall clock, so the jail may have contributed to the timeouts, but its contribution is unidentified. The number holds for the condition it was measured under and says nothing about Grok under a longer packet wall or without the jail.

The defects and the review

Three implementation defects surfaced during the run, and they threatened different things. The first two were the same bug in two coupled places. Output A, the worker score, pools the surviving packet runs of a cell and excludes only the ones that failed to complete, so failed packets force a cell to zero only when every packet fails. Output B, the synthesis, refused to run at all on any cell that had even one failed packet. On this run that split touched twelve of fifty six cells, and eight of them would have been handed a spurious zero on the synthesis side while their worker side scored normally, which would have biased the attenuation analysis. The asymmetry lived in both the synthesis step and the scoring step, and fixing either one alone would have left the same wrong answer while appearing clean, so both were corrected together to match the worker side's pooling exactly. The agent session running the benchmark caught this in validation, before any synthesis call was made.

The publication review caught the third. A grid in the scoring code filtered the task universe by condition but not by decomposition. The secondary decomposition, which exists for only two tasks, was therefore scored as though it spanned all twelve, with its pairwise statistics spread across ten tasks filled with zeros. That moved one secondary contrast from a correct 0.768 to a diluted 0.128. The defect corrupted only the secondary decomposition, and the primary decomposition, present for all twelve tasks, came out identical byte for byte after the fix, so the headline was never at risk from it.

The review itself ran in two rounds, each by a separate instance of GPT-5.6 Sol, the same model as the top scoring worker, reading the results against their source. It produced four corrections. They covered the significance overclaim, the causal story that put too much of Grok's failures on timeouts, the session's count of the biased cells (twelve where eight was precise) and the grid defect. The reviewer's verdict moved from publish with fixes to ready to publish only after each was addressed and a fresh pass confirmed that no new error had been introduced, and the full correction history is disclosed in the record.

The apparatus and its refreezes

The results rest on a record that a skeptic can check, and building that record took most of the agent session's time. The plan, the tool manifest and the answer key were each hashed and committed before the run. The outcome, the ledger's head hash, was committed after the matrix completed and before any score was computed, and the published numbers are bound to that head, with a negative control confirming that a wrong head is rejected. The gold corpus stays embargoed behind its hash, so the benchmark can be replicated without being memorized.

The apparatus went through five frozen versions, which makes four refreezes, and each is disclosed in the witness log. Two were availability failures that discarded partial matrices. In the first, one subscription hit a five hour session limit that killed 24 worker runs and stopped the matrix at 120 of 276. In the second, an authentication token died in flight and stopped it at 79, because the failover then in place did not recognize that second failure. The remedy was a failover that leaves no trace in the ledger: it treats both failures as availability events and retries the same run on a second subscription, leaving nothing scorable behind. A partial run cannot be repaired in place in a ledger built to resist tampering, so the apparatus discards and relaunches instead of patching. The other two refreezes were one fix for both coupled pooling defects, made in a single step, and one fix for the grid defect. The worker matrix that produced the headline ran before either of those fixes and stayed identical byte for byte through both, and its committed head hash, posted an hour before the first of them, shows that it predates them.

The full method, the intervals and permutation p values for each pair, the completion profile and the history of apparatus corrections are written up in a companion working paper that is not yet published. What the record supports, at the strength the preregistered test allows, is one resolved gap, GPT-5.6 Sol over Grok 4.5 on this plan and this corpus, with the rest of the ordering a point estimate that synthesis left unchanged.