What was being tested
Teknium of Nous Research announced on 2026-08-02 that Hermes Agent had become substantially more efficient, especially for smaller and weaker models, and attached an A/B card reporting roughly 21% fewer agent turns, 29% fewer tool calls, tool errors reduced to zero and 23% less wall clock on a weak model, with parity on a strong one.
The replication holds the model, the endpoint and the profile constant and varies only the harness version, then adds two other harnesses on the same models to answer a second question: how much does harness choice alone move cost per solved task.
Instrumentation
One counting proxy, not four sets of harness logs. Scraping each harness's own telemetry would have four definitions of a turn and four token conventions reporting into one table, so every cell routes through a small proxy that tallies each request against a run id in one schema regardless of which harness made it. The proxy also keeps real credentials out of per run configurations, since the harness sends a dummy key and the proxy attaches the real one server side.
Validation of the counter: on one model, the computed cost and the provider's own reported cost agreed exactly. Where an upstream reports its own cost, that figure is used in preference to a local price table, because it accounts for prompt cache discounts a flat table cannot see.
A protocol bridge, to avoid smuggling a serving confound back in. One harness speaks a different API dialect than the two cheap models offer. The available shim would have pointed it at a different vendor's serving of the same model while the others hit a broker, which is exactly the confound the proxy exists to close, so the proxy carries a translating route instead: messages, tool use and tool result blocks in one dialect on the way in, chat completions on the way out, and the event stream reconstructed on the way back. Every cell therefore reaches the identical upstream.
Minimal profiles. Every harness runs from an isolated uncustomised profile: fresh home directory, no user memory, no skills, no hooks, empty tool server configuration, fresh scratch working directory. This changes what the numbers mean and the change is deliberate. Comparing a bespoke twenty tool installation against a stock competitor would measure configuration and report it as harness.
The task set and its two guards
Twelve tasks in three families, built fresh for this run, each self contained with a verifier returning pass or fail:
- Family A, tool loop under induced error: a file the prompt misnames, a build script that fails its first invocation, a 4,000 line log whose answer sits at line 3,717, and a function that is called but never defined, where the correct answer is a refusal and the tempting one is a hallucinated path.
- Family B, multi step file and patch work: a field added across three files, a failing test to fix, a rename that must not break callers, and a three edit patch of which one edit is already applied, which punishes double application.
- Family C, long horizon build until a test passes: interval merging with touching versus gapped edges, column statistics over a CSV, an atomic ledger, and a tokenizer that must emit two character operators as single tokens.
Both guards ran before any paid cell. All twelve fail on their pristine seed, so nothing passes by doing nothing, and all twelve pass under a reference solution, so an unsolvable task cannot masquerade as a harness failure and poison a column. Graders live outside the scratch directory and pristine copies of every test script are restored at grade time, so a model cannot pass by editing its own test.
All thirteen cells
36 runs per cell, twelve tasks by three repetitions, 468 runs total in 43 minutes at concurrency six, for $3.03 against an $18 ceiling that was never approached.
| Cell | pass@1 | LLM calls | turns | tools | tool err | wall mean | wall med | tokens/solved | $/solved |
|---|---|---|---|---|---|---|---|---|---|
| Hermes 0.11.0 × Sol | 100% | sub | 5.0 | 5.0 | 1.8 | 30.5 | 31.0 | — | subscription |
| Hermes 0.19.1 × Sol | 100% | sub | 4.0 | 4.7 | 1.6 | 28.4 | 28.2 | — | subscription |
| Codex CLI × Sol | 100% | sub | — | — | — | 30.3 | 28.8 | — | subscription |
| Hermes 0.11.0 × Gemini 3.5 Flash-Lite | 100% | 8.3 | 8.9 | 9.7 | 4.8 | 16.5 | 15.3 | 90,152 | $0.02839 |
| Hermes 0.19.1 × Gemini 3.5 Flash-Lite | 97.2% | 6.4 | 6.5 | 6.8 | 2.7 | 10.4 | 9.9 | 87,355 | $0.02721 |
| Hermes 0.11.0 × DeepSeek V4 Flash | 100% | 4.6 | 4.6 | 5.5 | 1.9 | 34.7 | 27.7 | 50,765 | $0.00187 |
| Hermes 0.19.1 × DeepSeek V4 Flash | 100% | 4.5 | 4.5 | 5.4 | 1.7 | 27.5 | 25.6 | 61,417 | $0.00205 |
| OpenCode × DeepSeek V4 Flash | 100% | 5.9 | 4.9 | 5.9 | 0.1 | 47.6 | 56.0 | 104,603 | $0.00497 |
| Claude Code × DeepSeek V4 Flash | 100% | 4.8 | 6.7 | — | — | 38.7 | 36.7 | 72,658 | $0.00334 |
| Hermes 0.11.0 × Luna | 100% | 6.8 | 6.8 | 7.3 | 1.9 | 26.3 | 26.1 | 70,509 | $0.00257 |
| Hermes 0.19.1 × Luna | 100% | 5.7 | 5.4 | 6.8 | 2.1 | 59.6 | 21.7 | 90,063 | $0.00641 |
| OpenCode × Luna | 100% | 6.1 | 5.1 | 6.4 | 0 | 43.5 | 56.0 | 95,212 | $0.00384 |
| Claude Code × Luna | 100% | 6.3 | 8.9 | — | — | 32.9 | 32.0 | 89,583 | $0.00425 |
Subscription routed cells have no token accounting and no marginal cost. One harness reports turns but not a tool call count in its result envelope. Wall clock includes harness startup, which is structurally different per harness, so treat cross harness wall clock as indicative and the token and cost columns as the sound comparison.
Paired version delta
Same 36 task repetitions, same endpoint, same minimal profile, only the harness version differing.
| Model | LLM turns | tool calls | tool errors | median wall |
|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | −27.6% | −30.2% | −44.3% | −35.0% |
| GPT-5.6 Luna | −21.1% | −7.3% | +10.1% | −16.6% |
| GPT-5.6 Sol (strong control) | −19.0% | −7.2% | −9.4% | −9.0% |
| DeepSeek V4 Flash | −2.4% | −1.5% | −7.5% | −7.5% |
The published card gives 21%, 29%, tool errors to zero, and 23% on the weak model, with parity on the strong one. Two qualifications follow. The strong model row here is not parity, so the split reported upstream appears in this data as a gradient rather than a cliff. And DeepSeek breaks the pattern outright, gaining nothing behaviourally and spending 21% more tokens per solved task, so "weak models benefit" is too coarse a rule.
The runaway tail
| older version | newer version | |
|---|---|---|
| Runs emitting more than 10,000 output tokens | 0 / 36 | 3 / 36 |
| Total output tokens, that cell | 24,440 | 222,156 |
| Median output tokens per run | 793 | 587 |
| Mean wall clock | 26.3 s | 59.6 s |
| Median wall clock | 26.1 s | 21.7 s |
Three runs each contained a single API call returning exactly 65,536 output tokens, a hard cap being hit, over 341 to 643 seconds, while every other call in those same runs ran 61 to 284 tokens. All three passed, so this is a cost and latency failure that correctness metrics cannot see. It is what turns that cell's mean wall clock into a 127% regression while its median reads as a 16.6% improvement, and it accounts for 51% of the cell's spend. Reporting either the mean or the median alone would misinform the reader, so both are in the table.
Three events in 36 runs on one model pairing is a real rate with an undiagnosed mechanism. It may be a token ceiling default interacting with that model's reasoning output rather than anything about tool loops.
Cross harness, same model, same serving
DeepSeek V4 Flash
| Harness | pass@1 | tokens/solved | $/solved | vs best |
|---|---|---|---|---|
| Hermes 0.11.0 | 100% | 50,765 | $0.00187 | 1.00× |
| Hermes 0.19.1 | 100% | 61,417 | $0.00205 | 1.10× |
| Claude Code | 100% | 72,658 | $0.00334 | 1.79× |
| OpenCode | 100% | 104,603 | $0.00497 | 2.66× |
GPT-5.6 Luna
| Harness | pass@1 | tokens/solved | $/solved | vs best |
|---|---|---|---|---|
| Hermes 0.11.0 | 100% | 70,509 | $0.00257 | 1.00× |
| OpenCode | 100% | 95,212 | $0.00384 | 1.49× |
| Claude Code | 100% | 89,583 | $0.00425 | 1.65× |
| Hermes 0.19.1 | 100% | 90,063 | $0.00641 | 2.49×, tail driven |
Quality, and why none of it counts
467 of 468 runs passed. The single genuine failure was one repetition where the model never edited the buggy function at all. This ceiling effect was pre-registered as a risk before the matrix ran, with a stated bound that pass rate differences under about eight points would not be separable at twelve tasks by three repetitions; the real outcome was worse than that bound, since almost nothing failed. Do not read the pass rate column as a capability ranking. Harder tasks are the obvious next instrument.
The four artefacts
Three of the four measurement problems caught in this run were instrumentation masquerading as harness behaviour, and all three flattered a story the operator already believed.
- A 58% latency regression on the newer version. It was the proxy: it returned a not found response to a model discovery probe and the newer harness retried that probe twenty times. After serving the endpoint locally the two versions sat at 15.5 and 15.7 seconds.
- A multi minute startup penalty on one harness. It pays a database migration on every fresh home directory, and at 36 runs per cell that would have been charged to it as harness latency. The migration is now paid once into a template each run copies.
- A clean deterministic failure fingerprint: one harness failed the rename task in all three repetitions on both models, which is exactly the model independent harness signature the literature predicts. It was the grader. The task requires that no file still contain the old symbol, the grader greps the scratch directory, and that harness was the only one whose event transcript was written into that directory, quoting the model's own search for the old symbol. The harness had renamed everything correctly. Fixed two ways, by excluding instrument files from the grep and by keeping transcripts out of the graded directory, then all 468 runs were re-graded from their preserved scratch directories at zero API cost. Exactly six runs flipped, all of them these, and every flip is printed by the re-grading tool, because a silent re-grade is indistinguishable from moving the goalposts.
- A harness self reporting a cost two orders of magnitude above the measured one, because it prices another vendor's tokens at its own rates. Harness self reported cost is unusable on a redirected endpoint.
What the run does not settle
- Anything about quality, per the ceiling effect above.
- Anything about one vendor's own serving of one model. That vendor's API returned an authorisation error for the project in question, so those cells ran the same model through a broker, labelled as such. Serving is held constant across every harness being compared, so it is a constant rather than a confound, but these cells are not evidence about that vendor's serving.
- Anything about a customised installation. Every cell used a minimal profile. A separate measurement of a real customised install found a fixed preamble of roughly 16,000 tokens on every turn, around 7,000 of system prompt and 8,989 of tool schemas across 20 tools, before any task content reaches the model. That is a fact about that install rather than about the harness, and it is the untested lever most likely to matter: reducing tool count is testable locally and needs no version decision.
- Whether the runaway tail generalises.
Accounting check
The broker's own billing moved $2.64 across the session while the proxy independently attributed $2.588 to broker routed cells, the remainder being pre matrix pilot runs ledgered separately. Agreement within 2% against a third party's books is the strongest available evidence that the token and cost columns are real numbers rather than an artefact of a local price table.
The abort ceiling was unit tested rather than assumed: driving synthetic usage through the ledger fires it exactly at the threshold, after which every proxied call returns an error.