Model Identity Verification: Methodology and Raw Data

The question

A benchmark comparing Claude Opus 5 with GPT-5.6 Sol, run 2026-07-24 and 07-25, reported a result that reversed an earlier ranking. The challenge raised against it was whether the arm labelled Opus 5 had actually run Opus 4.8, since the opus alias on that machine resolved to claude-opus-4-8 at the time and the model generation was hours old when the benchmark started.

The question is not answerable from the launch scripts, because the launch script is the artifact under suspicion. It is answerable from the transcripts, which record a model field per assistant message, written per response rather than per invocation.

Method

Every row of the adjudication below was resolved from primary evidence: per-message model fields in session transcripts, resolved ids in the vendor CLI's own rollout logs, or run manifests. No row was resolved from a directory name, an arm label, or a summary line. All checks were read-only, and nothing in the experiment tree was modified.

The benchmark has three sub-experiments, referred to here as E1, E2 and E3. E3 is the one whose cells feed the reversed comparison.

Per-arm adjudication

Counts behind the table:

• E1 and E2 Opus arms: 1,105 cell working directories, 8,121 assistant messages, every one claude-opus-5. Zero messages carry claude-opus-4-8, claude-opus-4-7, or any other id.
• E3 Opus arm: 70 of 87 admitted cells retain durable transcripts, 401 messages, 100% claude-opus-5, zero drift. The other 17 rest on a ledger the gate wrote before a reboot cleared the transcript store.
• The judged subset: of the 30 Opus cells feeding the comparison, 13 have durable transcripts (66 messages, all claude-opus-5) and 17 are ledger-grade. The set difference between those 30 cells and the admitted 87 is empty.
• Control for instrument sensitivity: a 250-file sample of July transcripts on the same machine contains six distinct model ids, including both claude-opus-4-8 and claude-opus-5, so the field discriminates the two generations rather than reporting a constant.

The gate

The frozen design states the hazard in writing: the opus alias resolves to claude-opus-4-8 on that box, so the explicit id is always passed. The scoring rule built on that statement admits an Opus cell only when a transcript exists and every message field in it names the intended model, and rejects the cell otherwise.

The rule fired against its own experiment. It marked 141 cells unverified and excluded them from every reported number, which is why the admitted count is 87 rather than 228.

Three things kept the benchmark clear of the trap. The explicit model id was passed at all three launch sites and the alias was used at none of them. The wrapper script the Opus arm runs through does not rewrite the model argument: it sets a config directory, strips API key, auth token and base URL from the environment, and executes the CLI with the caller's arguments intact, so an explicit --model reaches the CLI where it outranks any settings default. And the gate refused unproven cells rather than assuming them.

The asymmetry the check exposed

The gate's identity rule applies to Opus cells by construction. Every Sol cell carries the literal string n/a in the gate's model column and is admitted as valid on the strength of its launch line, so the benchmark's summary claim that the Sol arm carried the stronger provenance is inverted at the identity layer: the Opus arm is the only arm whose identity the instrument proved.

Sol identity is independently corroborated for the run window by the Codex CLI's own rollout logs, which contain gpt-5.6-sol and no other model id across the two run days. That evidence is external to the benchmark. The correct fix for the next benchmark in this family is to parse the resolved model from each rollout file per cell, which is the same one-field check the Opus arm already gets.

Timeline corroboration

The earliest message carrying claude-opus-5 anywhere in the transcript store is 2026-07-24T17:46:55Z. The benchmark's first Opus cell transcript begins at 2026-07-24T18:04:23Z, roughly eighteen minutes later, which is consistent with a model that had just become available.

The corollary is a dating rule for the whole machine: any artifact older than 17:46:55Z on 2026-07-24 that refers to "opus" refers to 4.8, whatever the label says.

Limitations

1. Seventeen E3 cells are ledger-grade, not transcript-grade. Their evidence is a file the gate wrote at run time, not a durable transcript, because a reboot cleared the store. The ledger is a weaker instrument than the transcript it was derived from.
2. The check proves the model, not the account. The per-account transcript directories on that machine are symlinks into one shared store, so the transcript belonging to a cell proves which model produced it and does not prove which subscription was billed for it.
3. The semantic verifier used downstream was not identity-checked, and no per-call record was kept. It is arm-blind by construction, so the omission does not favour either arm.
4. A model id is not a system. Identity verification says which weights answered. It says nothing about reasoning effort, harness version, or tool surface, and one of those is a live confound in this benchmark: the arms ran at different reasoning efforts, which is why the comparative result is withheld from the parent post pending a matched re-run.
5. Nothing here was re-run. This is a verification pass over artifacts that already existed, with the failure mode being that an artifact absent from disk cannot be adjudicated at all.

Reproduction

The checks are four greps and a set difference. Against an experiment tree laid out as this one is, with E set to the experiment root:

# 1. Launch-site model ids for every arm
grep -n "model claude-opus-5\|-m gpt-5.6-sol" $E/e{1,2,3}/run-e{1,2,3}.sh

# 2. Per-message model fields, Opus cells of E1 and E2, independent of the gate
for d in $E/e1/work/*-opus5-* $E/e2/work/*-opus5-*; do
  p="$TRANSCRIPTS/$(readlink -f "$d" | sed 's#/#-#g')"
  [ -d "$p" ] && cat "$p"/*.jsonl
done | jq -r 'select(.message.model != null) | .message.model' | sort | uniq -c
# -> 8121 claude-opus-5, nothing else

# 3. E3 durable transcripts
cat $E/e3/transcripts/*/*.jsonl \
  | jq -r 'select(.message.model!=null)|.message.model' | sort | uniq -c
# -> 401 claude-opus-5

# 4. Gate ledger by arm
awk -F'\t' 'NR>1{a=($1~/opus5/)?"opus5":"sol"; print a"\t"$6"\t"$7}' $E/e3/e3-gate.tsv \
  | sort | uniq -c
# -> 87 opus5 VALID-PREREBOOT / 141 opus5 UNVERIFIED-MODEL / 78 sol n/a VALID

# 5. Judged Opus cells are a subset of the admitted cells
comm -23 <(jq -r 'select(.arm=="opus5")|.cell' $E/e3/d6/joined.jsonl | sort -u) \
         <(awk -F'\t' 'NR>1 && $7=="VALID-PREREBOOT"{print $1}' $E/e3/e3-gate.tsv | sort)
# -> empty

The equivalent check for a Codex arm reads the resolved id out of the CLI's rollout files for the run window rather than trusting the -m flag.

What the pass changed

Nothing was re-run on identity grounds, since no number in the benchmark was produced by the older model while labelled as the newer one. What the pass added is a gate for the arm that lacked one, and a correction to the benchmark's own provenance wording, which had described the unverified arm as the better evidenced of the two.