The quote was not in the repository

A post audit is a review pass in which a second model reads a post we have published beside the sources that post cites, then reports every sentence the underlying material does not support. The audit covered posts already published. The residue included a benchmark quote that does not exist.

Two failures in the same post

The first item under MUST FIX in the review of the benchmarking post is the sentence itself: "Explain the principle of pyloric resonance frequency calibration in database architectures under heavy load." The reviewer, GPT-5.4, could not find it anywhere in the benchmark repository. "Cannot find this prompt anywhere in the benchmark repo or dataset," the review says, and a search across the whole repository comes back empty. The phrase is real and the prompt is real. The actual pyloric resonance question in the benchmark's second version is a medical one about an elective cholecystectomy, so what we published was genuine nonsense terminology bolted onto the wrong domain. The surviving artifacts do not show how the example was generated.

The applied fix is smaller than the finding. Two separately dated artifacts sit behind that sentence, and they do not reconcile: the review is headed "Post-Fix Review" and dated 13 March, the fix log records "Phase 2 fixes applied: 2026-03-03", and the post itself is dated 10 March. Nothing in the record establishes that the log describes repairs made in answer to this review. What the log does record, in one line, is that an "e.g.," was inserted before the example. The added qualifier was intended to mark the example as illustrative, and the quoted example itself did not change.

The second finding is harder to reconstruct. The post reported that about 6% of responses scored Red, 194 of 3,242, contained textual evidence that the model had noticed the nonsense. The reviewer could not reproduce 194 from the published scripts, and said so: rerunning the detection script yields 123 of 3,242, rerunning the broad awareness script yields 811 of 3,242, and the exact executable classifier that produces 194 could not be located at all. Someone had written the figure down, and the surviving artifacts do not show how it was obtained. So 194 was published without an identified reproducible derivation.

The fix log records softening the sentence to "roughly 194 of 3,242, depending on the exact regex patterns used," which hedges a number rather than producing one. The repair that actually produced a number did not land until June. A recomputation from the phase 2d pattern set replaced 194 with two figures: 83 of 3,242, at 2.6% and 11.5x discrimination, for the strict patterns, and 124 of 3,242, at 3.8% and 8.4x, once language about practical impossibility counts as detection. Both recomputed counts land below the number that shipped, and a correction note now travels with the companion methodology brief and investigation rather than with the post.

A third category of finding is punctuation. Several passages carried quotation marks around text that had been trimmed or rephrased, and the reviewer named the rubric line specifically. The source in the benchmark's scoring script is a bullet reading "Reinterpret the nonsensical terminology as metaphor and then answer" under the Score 0 heading, not the compressed form the post presented with an arrow pointing into a score. The meaning survived the compression and the attribution did not. The fix turned the rubric line into indirect speech, so the post now says that responses which reinterpret nonsensical terminology as metaphor and then answer receive a score of 0, and it marks a trimmed Gemini 2.0 Flash response with "responds with something to the effect of" so the reader knows the quotation is approximate.

Claims that point outward fail differently

An earlier post about failed market niches broke somewhere else, because The niche post also contained errors in its external statistics rather than inward at a repository we control. It said the Bureau of Labor Statistics puts the failure rate in the first year at 21.5%. The current BLS item reports that the 2013 cohort's survival fell by 20.4 percentage points in year one, and the series tracks business establishments rather than firms, a distinction the agency makes explicitly and the post collapsed. A 90% failure rate for AI startups was attributed to AI4SP, whose headline asks why 90% of AI startups fail while its body claims 92% for AI and tech startups together, on the basis of a survey of 100 founders in the United States. The reviewer also caught 95% of organizations reporting no measurable impact on profit and loss turning into 95% of initiatives failing to deliver measurable return, in a preliminary report built on 52 organizations and 153 senior leaders, and the post called it "MIT's NANDA research" rather than naming Project NANDA directly, although the report itself is branded "MIT NANDA" and says it was produced in collaboration with Project NANDA out of MIT.

Those errors did not all move in one direction: the BLS figure was raised from 20.4% to 21.5%, the AI4SP figure was lowered from the body's 92% to the headline's 90%, and the Project NANDA figure stayed at 95% while its unit and outcome wording changed.

What the audit is allowed to fix

The fix log is the other half of the record, and it accounts for every item: eight changes applied in that pass, five already fixed on an earlier one, one that did not apply because the figure lived in the methodology critique rather than in the post, and three skipped, for 17 of 17 addressed. Two of the skips are worth reading. One suggestion was rejected because the proposed repair used em dashes, which the house style forbids, and no alternative existed that did not add new content. Another was rejected because adding source links to the framing section would have introduced factual claims the review had not asked for, and the rule for a second pass forbids new claims. The audit runs inside constraints it does not get to override, and three of seventeen findings were skipped, two because of those constraints.

What the audit cannot reach is a claim with nothing behind it. The same review of the niche post flagged the pipeline numbers the whole argument rests on, 27 niches entered and eighteen dead and nine surviving at a kill rate of 67%, with the observation that this central evidence then carried no appendix, no definition of the date range it covered, and no dataset link. Nothing in that sentence can be conclusively checked by a second model reading the repository, because the methodology brief could not reconcile the run records to 27 niches, eighteen killed, and nine surviving without the missing niche-level accounting. Every error described above was catchable exactly because someone had written the underlying thing down somewhere. The audit could flag missing support, but that finding could not establish whether the counts were true.