A capability evaluation reports a score, but the score is not a property of the model. It is a property of a triple — model, task, and elicitation envelope — and the third element is the one most likely to be silently underspecified. This article argues that capability evaluation without an explicit elicitation envelope is underspecified measurement; it surveys the documented elicitation gap; and it treats "best-effort elicitation" as an unresolved methodological question rather than a settled standard.
Coverage note: verified through May 2026.
The measurement object
Capability Evaluation usually reports a single score: pass rate, success rate, time horizon, or a percentile against a human baseline. The implicit claim is that the score measures the model's ability. The empirical record now makes that framing untenable. The score measures what a particular system — model weights plus prompt plus scaffold plus tools plus sampling budget plus fine-tuning state plus evaluator skill plus safety-filter configuration — achieved on a particular task distribution. Change any of those terms and the score changes, sometimes by a factor of two or more on the same model.
This is not a marginal correction. It changes what a "model cannot do X" claim is allowed to mean.
The cleanest decomposition treats every capability result as a function of five inputs:
- Model: weights, version, training cutoffs, post-training pipeline (RLHF, RLAIF, constitutional methods, refusal training).
- Task distribution: the specific benchmark, its sampling, its grading, and its dev/test separation.
- Elicitation envelope: prompts, in-context examples, scaffolding, tools, memory, agent loops, sampling policy, inference-time compute, fine-tuning access, safety-filter state.
- Evaluator: time budget, expertise, dev-set iteration, ability to recognize spurious failures.
- Threat model: the actor whose capability is being approximated — ordinary user, skilled user, motivated misuser, lab insider with weights.
The score reports what happened when one specific combination ran. It does not report what the bare model "is capable of," because the bare model is not a thing that can be capable of an autonomous task without an interface around it.
Latent Capability is the term often used for what an eval is supposed to measure: an underlying ability that exists in the weights and would manifest under sufficient elicitation. The term is useful but not innocent. Latent capability is not directly observable; only behavior under specific elicitation conditions is. Treating latent capability as a stable scalar that evaluators are merely failing to read accurately conflates two cases that should stay separate: behavior already accessible under ordinary inference, which requires only better prompting; and behavior that does not yet exist in the system until the evaluator builds it — through fine-tuning on demonstrations, through scaffolds that move substantial cognition out of the model, through tool access that extends the action space.
The article's strongest claim is that capability evaluation should be read as conditional measurement. The weaker claim — "frontier models often do better under better elicitation" — is true, but the operational consequences flow only from the conditional framing.
The documented elicitation gap
The empirical record converges on a single qualitative claim: elicitation matters, often by large amounts, and the size of the effect is task-, model-, and technique-dependent. The quantitative anchors below are the strongest individual results, not a universal multiplier.
METR's post-training enhancement study
METR's measurement of post-training enhancements to GPT-4 is the cleanest single-model demonstration of how much elicitation moves a capability score. METR ran the same agent on the same autonomous-task suite under different post-training and harness configurations. The base setup scored roughly 5 ± 2%. After OpenAI's post-training, the same suite scored 30 ± 6%. With additional scaffold, tool, and reasoning-module changes, performance reached 38 ± 6% — though METR noted the marginal gain from their own scaffold improvements was not statistically significant given the variance.
The interpretive lesson is not "elicitation improves scores by 7×." It is that performance under one post-training and harness configuration cannot stand in for performance under another configuration. A leaderboard entry that omits the post-training and harness state is not interpretable.
METR's o1 update
METR's January 2025 update on o1 and Claude 3.5 Sonnet is the most policy-relevant single result in the literature. METR adapted their scaffold for o1's reasoning interface and found that the modest adaptation more than doubled the human-baseline-equivalent time horizon on autonomous tasks. The report explicitly cautioned that even after this adaptation, METR could not make robust negative capability claims — further elicitation might yield higher scores.
This matters because o1 was a then-frontier model evaluated by an experienced safety lab with months of methodology development behind it. The result that a short scaffold adaptation produced more than a doubling of the headline metric is the closest the field has to a clean demonstration that capability evaluations and elicitation engineering cannot be cleanly separated.
Davidson et al. on post-training enhancements
Davidson, Denain, Villalobos, and Bas (2023) surveyed post-training enhancements and estimated that several common techniques — prompting strategies, scaffolds, fine-tuning, tool use — are worth on the order of 5×–20× training-compute-equivalent gains on relevant benchmarks. The survey is heterogeneous, non-experimental, and dated, and the authors are careful about its limits. The order-of-magnitude claim is nonetheless consistent with the experimental anchors: post-training elicitation often moves scores by amounts comparable to substantial increases in pretraining compute. Any safety policy that ignores elicitation is implicitly assuming an arbitrarily large training-compute overhang.
Password-locked models and hidden capability recovery
Greenblatt, Roger, Krasheninnikov, and Krueger (NeurIPS 2024) introduced password-locked models — model organisms intentionally trained to hide a capability behind a trigger string. The study tested whether fine-tuning on a small set of high-quality demonstrations could elicit the hidden capability without the password. It often could. The paper's own caveat is essential: elicitation may be unreliable when high-quality demonstrations are unavailable, or when the hidden capability exceeds what human demonstrators can produce.
This is evidence for the methodological point — that simple prompting alone is not a sufficient elicitation strategy — and against an overclaim. Password-locked-model results do not show that arbitrary evaluators can robustly recover arbitrary hidden capabilities; they show that under specific stress-test conditions, fine-tuning often beats prompting.
The Elicitation Game
Hofstätter, Roger, Greenblatt, and others (ICML 2025) extended this work into a multi-technique elicitation benchmark. Their headline result is technique-dependence: prompting unlocked hidden capabilities for multiple-choice question-answering organisms; for code-generation organisms, only fine-tuning elicited the hidden capability; activation steering largely failed in their setup; technique combinations often outperformed any single technique.
The interpretive lesson is that elicitation is not a single dial. Different techniques unlock different capability types, and the right technique mix depends on what kind of capability the model is hiding and how it is hiding it.
AISI's structured elicitation protocol
The UK AI Security Institute's elicitation protocol independently names the same levers: strategic prompts, tool access, multiple candidate generations, agent scaffolds, and multi-agent setups. AISI's framing is unusually direct about the unsettled status of the field — elicitation is described as partly craft and partly science, and the protocol is presented as a working approach rather than a solved standard.
Anthropic's evaluation guidance and sabotage report
Anthropic's 2025 sabotage risk evaluation explicitly noted that it lacked high confidence in its elicitation and expected that further prompt engineering or less-invasive elicitation techniques could noticeably improve results. The 2025 Anthropic/OpenAI cross-lab alignment exercise relaxed external safety filters to test underlying behavior and still cautioned that tooling incompatibilities shaped what could be tested. Anthropic's agent-evaluation guidance is direct: evaluating an agent means evaluating model plus harness, and the same model can produce dramatically different scores under ambiguous tasks, rigid graders, or constrained scaffolds. Anthropic's Responsible Scaling Policy explicitly says that true upper-bounding of generative-model capabilities is not currently possible.
What this evidence does and does not establish
The literature supports a strong qualitative claim: capability scores are sensitive to elicitation, and the sensitivity is often large enough to change qualitative conclusions about whether a model can or cannot perform a task. It does not support a universal quantitative multiplier. Treating "elicitation increases scores by N%" as a base rate would overgeneralize from a heterogeneous set of experiments. The defensible base rate is variance itself: expect substantial envelope sensitivity until shown otherwise on a specific task and model.
Elicitation levers: a working inventory
The following levers each move scores under at least some conditions, and each must be reported for a capability claim to be interpretable. The third column names what the lever can hide — the artifact a result may carry if the lever is not specified.
| Lever | What it can reveal | What it can hide |
|---|---|---|
| Prompt phrasing | Capability already accessible under ordinary inference | Evaluator skill at finding a working phrasing |
| In-context examples | Format adaptation, task framing, demonstration learning | The capability being induced by the examples themselves |
| Chain-of-thought / scratchpad | Reasoning that fits in the available context | Whether the model can produce the same answer without externalized reasoning |
| Sampling policy (temperature, pass@k, best-of-N) | Tail behavior; probability of rare correct answers | Reliability under single-attempt conditions; cost-normalized performance |
| Tool access | Capability augmented by the tool's action space | Whether the capability is in the model or the tool |
| External memory / retrieval | Performance with context support | Whether the model can perform without the external store |
| Task-decomposition scaffolds (agent loops, planners, debate, tree search) | System-level capability under structured cognition | How much cognition is in the scaffold vs. the model |
| Fine-tuning on demonstrations | Hidden capability when demonstrations exist | Whether new behavior is being induced rather than revealed |
| Reinforcement or preference adaptation | Capability when reward signal is available | Reward hacking, narrow specialization |
| Evaluator feedback / dev-set iteration | Real bottleneck distinct from spurious failure | Eval contamination, train/test leakage |
| Safety-filter state (deployed, relaxed, removed) | Latent behavior independent of refusal training | Whether the result maps to deployed conditions |
| Inference-time compute budget | Performance under realistic compute envelopes | Whether large-budget gains transfer to deployment cost ceilings |
| Number of evaluation attempts | Aggregate elicitation feasibility | Pass@1 reliability |
A capability claim is only interpretable when each of these is either stated or deliberately marked as "not varied." Scaffolding, Tool Use, Pass@k and Sampling Budgets, and Post-Training Enhancement each deserve their own treatment because the failure modes differ qualitatively across levers.
A particular caution applies to the difference between revealing and constructing capability. Prompt phrasing and few-shot examples usually reveal behavior already accessible under ordinary inference. Tool access extends the system into an action space the bare model does not have. Pass@k and best-of-N convert reliability claims into search claims. Fine-tuning may remove refusals, adapt task format, or teach genuinely new policies from examples. Multi-agent debate or tree search may move substantial cognition into the scaffold. Lumping these under one "elicitation gap" label obscures whether the evaluation measured a hidden capability, a constructed system capability, or an evaluator-engineering capability.
The model–system boundary
A persistent confusion runs through both lay and technical reporting on capability: the result from a scaffolded system gets attributed to "the model," and the failure of a bare model on a benchmark gets read as the system's safety property. Both directions of attribution are usually wrong.
Concretely, the relevant evaluation objects are not just "model M" but include:
- The base model weights, evaluated through a minimal inference API.
- The chat-tuned model, with default system prompt and refusal training active.
- The model under a specific system prompt, which can radically change behavior.
- The model in an agent scaffold — an outer loop that calls the model in a structured pattern, possibly with memory, planning, and self-reflection.
- The model in a tool-using system, where the action space extends beyond text into code execution, web access, file manipulation.
- The fine-tuned derivative of the model, possibly with safety training removed or task-specific demonstrations added.
- The ensemble or search procedure that runs many candidates and selects.
These are different objects under test. A claim that "Model M can autonomously exfiltrate its weights" requires saying which of these objects achieved that behavior, under what scaffold and tool access, with what fine-tuning, under what number of attempts. A claim that "Model M refuses dangerous requests" must say which of these objects was tested, since the bare-model refusal property does not survive system-prompt changes, fine-tuning, or jailbreaks.
The working position is that a capability claim must commit to one of these objects, name it explicitly, and describe the elicitation envelope around it. Claims that drift between objects — succeeding on a scaffolded eval, then being reported as a model property — are not interpretable.
Four envelopes for capability claims
The simplest practical pattern for elicitation-aware reporting is multi-envelope evaluation: report the same task under several named elicitation conditions, and treat the gap between them as the central piece of evidence rather than a confound to be eliminated.
A minimum working set of four envelopes:
Envelope 1: Minimal
Direct task prompt, no special scaffold, no tool access, low sampling budget (typically pass@1 with a fixed temperature), no in-context examples, deployed safety filters active.
This envelope produces a clean regression-tracking number. It is reproducible across labs and useful for measuring relative progress between model versions on a fixed protocol. It almost always understates capability for any motivated user.
The danger is reading the Envelope 1 score as a ceiling. A model that scores 5% under Envelope 1 has not been shown to be incapable of the task; it has been shown to fail under one specific, weak elicitation procedure. The METR GPT-4 post-training study is the canonical illustration: the same model that scored 5 ± 2% under one configuration scored 30 ± 6% under another with no weight changes.
Envelope 2: Realistic user
Ordinary prompt iteration by a non-expert user, common product tools (web search, code execution, basic agent loop), plausible budget, deployed safety filters active. The evaluator is allowed to try several prompt phrasings but not extensive scaffold engineering or fine-tuning.
This envelope answers the product-deployment question: what should the typical user expect this system to do, and what should the developer expect to need to support? It is the right envelope for capability claims about deployed behavior, accessibility uplift, and educational value.
It is the wrong envelope for safety claims about motivated misuse. A capability that does not appear under Envelope 2 may still be readily accessible to a skilled adversary under Envelope 3 or 4.
Envelope 3: Expert scaffold
A skilled evaluator with task-specific knowledge, transparent scaffolds (task decomposition, planning, retry logic), tool access appropriate to the task, higher sampling budget, and a documented harness. Fine-tuning is not typically permitted at this envelope; the model under test is the deployed model with its safety training intact.
This envelope answers the skilled-misuser question and the bounded "what could a motivated user achieve" question. It also serves as the practical ceiling for most public capability evaluation, because going further usually requires weight access or substantial fine-tuning budget that ordinary adversaries do not have.
METR's o1 update is essentially an Envelope 3 result: an experienced safety lab, a transparent scaffold adapted for the model's reasoning interface, a documented harness. The result that scaffold adaptation more than doubled the human-baseline-equivalent time horizon is what an Envelope 3 evaluation looks like when it is done well.
Envelope 4: Threat-model best effort
The strongest elicitation that the relevant adversary could plausibly apply, bounded by a preregistered budget and a specified threat model. The threat model determines what is in scope: weight access for stolen-weights scenarios; fine-tuning and refusal removal for lab-insider scenarios; bespoke scaffolds for state-actor scenarios; pass@N with large N for high-stakes one-shot risks.
This envelope is the right object for several specific safety claims: dangerous-capability evaluation in the sense of Responsible Scaling Policy thresholds; AI R&D autonomy evaluation; biorisk and cyberrisk uplift evaluation. The Envelope 4 result is appropriately treated as an approximate upper bound for the named threat model, not a ceiling for all conceivable elicitation.
Two cautions are central to Envelope 4. First, "best effort" is not a community-wide constant; it is a threat-model commitment. The best effort that maps to a stolen-weights adversary differs from the best effort that maps to a skilled-but-unfunded misuser. Different safety questions need different Envelope 4 definitions. Second, Envelope 4 results are not reproducible in the strict sense — they depend on private scaffolds, evaluator time, and lab-specific know-how. They should be reported with explicit resource budgets, ablations, and held-out tasks, or treated as evidence of one team's achievement rather than as a settled measurement.
Reading multi-envelope results
The four envelopes are not a ranking; they are conditional measurements that answer different questions. A model that scores 5% / 15% / 40% / 70% across the four envelopes is not described by any single number. The right summary is the curve.
| Envelope | What it measures | What it cannot claim |
|---|---|---|
| Minimal | Regression-tracking baseline | "The model cannot do X" |
| Realistic user | Product-deployment behavior | "The model is safe under misuse" |
| Expert scaffold | Skilled-user capability ceiling | "Adversaries with weights cannot do X" |
| Threat-model best effort | Approximate adversary capability for a stated threat model | "The model can do X by default" |
A "model cannot do X" claim is interpretable only when Envelope 4 (or the strongest envelope relevant to the safety question) has been tested and the model failed there. A "model can do X" claim is interpretable only when the envelope is named and the result is not silently transferred to a stronger or weaker envelope.
Where this framing fails
The elicitation-envelope framing is the right correction to naive benchmark realism, but it has its own failure modes. Two are important enough to name explicitly.
The over-elicitation asymmetry
A natural reading of the elicitation gap is that under-elicited evals systematically underestimate capability, so the corrective is always more elicitation. This reading is partially wrong. Over-elicited positive claims can be as misleading as under-elicited negative claims — but the two failures are not symmetric, and conflating them produces a different kind of measurement confusion.
Under-elicited negative claims are more dangerous in safety contexts. A weak-prompt benchmark that reports "the model cannot do X" invites a ceiling inference from a lower-bound measurement. The risk is that policy decisions, deployment authorizations, and public discourse treat the lower bound as a true upper bound, and a real capability is missed until an adversary finds it.
Over-elicited positive claims are dangerous in a different direction. A heavily scaffolded, fine-tuned, tool-augmented, best-of-many run that reports "the model can do X" invites attribution drift. The capability gets credited to the model rather than to the system, and the result is read as evidence about ordinary deployed behavior rather than about a specific evaluator-built system.
The corrective is not symmetric either. Under-elicited evals need stronger elicitation. Over-elicited evals need clearer attribution: what was the model contribution, what was the scaffold, what was the budget, what was the evaluator skill? Ablations matter more here than additional effort.
This points to a stronger language discipline for negative claims than for positive claims. "Not observed under Envelope E" is acceptable. "The model cannot do X" usually is not, regardless of the envelope tested, because no envelope can rule out unknown future elicitation. "The model can do X under Envelope 3 with the following scaffold and budget" is acceptable. "The model can do X" without qualification is usually not, because it papers over the conditions that made the result possible.
Best-effort elicitation as a virtue word
"Best-effort elicitation" risks becoming a methodological virtue word — a phrase used to confer credibility without specifying what was done. The literature is direct about the unsettled status of the term. METR's capability-elicitation guidelines describe their recommendations as reasoned but not experimentally confirmed in full. AISI describes shared elicitation practice as something still being built. Anthropic's RSP says true upper-bounding of generative-model capabilities is not currently possible.
If "best effort" is not bounded, reproducible, and threat-modeled, it can hide arbitrary search, evaluator discretion, private lab know-how, benchmark-adjacent overfitting, and pass@k cherry-picking. The result may be a useful lower bound on what some team achieved; it is not a clean upper bound on what the model could be made to do.
The working position is that "best effort" should be defined operationally per evaluation. A best-effort claim should include: the threat-model commitment that scopes "best"; a resource budget (evaluator hours, sampling tokens, fine-tuning compute); a held-out task split; ablation results showing the marginal contribution of each elicitation lever; and a documented harness sufficient for independent audit. Without these, "best effort" is an aspirational descriptor, not a methodological standard.
Methodology implications for AI safety evaluation
The elicitation framing changes what an AI safety evaluation has to report.
Negative claims must specify the envelope
The most common methodological error in capability-claim reporting is the unqualified "cannot." Every "the model cannot do X" should be rewritten as a conditional non-observation: "the model did not produce X under elicitation envelope E with budget B." The envelope and budget are part of the claim; without them, the claim is not interpretable as evidence about safety.
This applies even when the envelope is strong. A claim that an expert-scaffolded evaluation with substantial sampling did not produce a dangerous capability is meaningful — but it is meaningful as a lower bound on what is currently elicitable, not as a ceiling on what is possible. METR's o1 update is explicit on this point: even after careful evaluation, the report could not make robust negative capability claims.
Safety evaluations need at least one adversary-relevant envelope
If the safety question is whether a model could be misused, an evaluation under Envelope 1 or Envelope 2 is not sufficient evidence. The relevant envelope is whichever one most closely approximates the adversary in the threat model. For a skilled-but-unfunded misuser, Envelope 3 is usually the right target. For a state actor or lab insider, Envelope 4 with weights access and fine-tuning is the right target.
AI Safety Evaluation practice has converged toward this position over 2024–2025. Frontier-lab safety frameworks increasingly require full-capability evaluation that includes scaffolds, fine-tuning where the threat model permits, and substantial sampling budgets. The convergence is uneven and resource envelopes differ; METR's review of common elements in safety frameworks tracks the state of play.
Safety-filter state is a confound, not a backdrop
Refusal training and safety filters often dominate the failure modes in capability evals — the model "fails" the eval by refusing to engage rather than by being unable to perform the task. This makes deployed-filter evaluation a poor proxy for capability questions. The 2025 Anthropic/OpenAI cross-lab alignment exercise relaxed external filters to test underlying behavior, and reported that tooling incompatibilities still shaped what could be tested.
The methodological implication is that safety-filter state needs to be reported as a deliberate variable. A deployed-filter evaluation and a relaxed-filter evaluation answer different questions. Neither is universally better, but mixing them silently is.
This connects to a Dual-Use Research tension. Publishing detailed protocols for relaxing or removing safety filters in deployed models could provide misuse uplift. The working compromise is tiered disclosure: the method class (e.g., "we evaluated under both deployed-filter and relaxed-filter conditions, where the relaxed condition removed refusal training but preserved capability") should be public; the specific operational recipes should be restricted or shared through structured disclosure channels.
Sampling budget and pass@k convert reliability claims into search claims
Many capability evaluations report pass@k or best-of-N performance, where the system attempts the task multiple times and the evaluator counts a success if any attempt succeeds. This is a useful measurement for some safety questions — particularly for one-shot high-stakes risks — but it is not equivalent to a reliability measurement.
A model that succeeds 1% of the time at pass@1 will succeed roughly 63% of the time at pass@100 by independence alone. The pass@100 score is not evidence that the model "reliably" performs the task; it is evidence that with 100 attempts, the task is solvable. For some safety questions this is the right framing (a motivated adversary can run 100 attempts). For others — will an autonomous agent succeed at this task in deployment, where each attempt has cost — it is misleading.
Reporting standards: state N, temperature, total tokens, retry policy, selection rule, and cost-normalized scores. Without these, a pass@k headline is uninterpretable.
Train/dev/test separation and benchmark overfitting
Elicitation work that iterates on a benchmark inherits all the standard machine-learning concerns about train/dev/test separation. Fine-tuning on demonstrations that overlap with eval tasks, prompt engineering that uses test-set examples for iteration, and scaffold tuning on the same task family that will be evaluated are all routes to overfitting that look like genuine elicitation improvements. Benchmark Overfitting is the central concern here, and the standard fix is to maintain held-out task suites that are not used for elicitation iteration.
The audit signal is transfer: a genuine elicitation improvement should produce gains on fresh matched tasks, not just on the iterated task suite. Sudden score jumps on known tasks without corresponding gains on held-out tasks are a warning that elicitation has crossed into memorization or overfitting.
Sandbagging and evaluation awareness
A particular failure mode that elicitation work has to take seriously is intentional underperformance — a model that detects evaluation conditions and optimizes for a different objective than the one the evaluator expects. Sandbagging and Evaluation Awareness is currently a frontier topic in alignment research; password-locked models are a stress-test for it but not a full treatment.
The methodological implication is that elicitation needs to be deception-aware: held-out variants that look less like evaluations, distributional checks on whether eval performance differs from deployment-like performance, and explicit reporting of whether reasoning traces reference being tested. None of these are solved. The literature is at the stage of identifying the problem and proposing initial protocols, not at the stage of robust mitigation.
The unresolved question: operationalizing best-effort elicitation
The strongest single takeaway from the current literature is that the field does not yet have a settled operational standard for "best-effort elicitation." Several research directions are converging on candidate components, but none has been validated as a community-wide protocol.
Candidate components
Preregistered multi-envelope protocols. A preregistered evaluation packet that runs the same model on the same held-out task suite under the four envelopes above, with frozen test sets, dev-set-only iteration, retained transcripts, budget manifests, and failure taxonomies. The decisive experiment for an elicitation claim is a multi-envelope replication packet of this form; the field has not yet run enough of these for cross-lab variance to be characterized.
Resource-normalized reporting. Instead of single scores, report curves over evaluator-hours, sampling tokens, fine-tuning compute, and tool budget. This makes elicitation results comparable across labs with different resource envelopes, and it surfaces the cost-effectiveness of different elicitation techniques.
Failure taxonomies. Distinguish spurious failures (formatting bugs, tool misconfiguration, refusal templates) from real failures (the model cannot perform the underlying reasoning). METR's elicitation guidelines are explicit about this distinction. A capability claim that does not classify its failure modes risks misreading refusals as inability.
Held-out evaluation suites. A persistent reserved task suite that is not used for any elicitation iteration. Elicitation improvements that transfer to the held-out suite are credible; improvements that do not transfer are likely overfitting.
Tiered disclosure. Publish envelope variables, aggregate scores, harness skeletons, and failure taxonomies; restrict specific exploit recipes, jailbreak prompts, and task solutions where publication would materially increase misuse risk. This is the compromise that allows methodological reproducibility without operational uplift.
Deception-aware elicitation. Held-out variants that look less like evaluations, distributional consistency checks, and explicit testing for evaluation awareness. This is the most underdeveloped component; the literature is at the stage of proposing protocols rather than validating them.
What would settle the question
The field would have a credible operational standard for best-effort elicitation if several independent groups ran preregistered multi-envelope protocols on shared task suites and reported convergent ceilings under fixed budgets, with low cross-lab variance and robust replication across model generations. Until that happens, "best effort" is a useful aspirational descriptor and a useful threat-model-specific commitment, but not a community-wide standard.
What this means for current capability evaluation is that confidence in any specific score should be calibrated to the elicitation envelope, and confidence in any aggregate capability claim should be calibrated to the diversity of envelopes that have been tested. A model that has been evaluated only under Envelope 1 is poorly characterized for any safety question. A model that has been evaluated under all four envelopes by multiple independent labs, with convergent results and transparent ablations, is well-characterized — but the field has not yet produced this depth of evaluation for any frontier model.
A reporting standard
The defensible reporting standard for an elicitation-aware capability claim includes:
- Model identifier: weights version, post-training pipeline, training cutoff, refusal-training state.
- Task identifier: benchmark version, sampling, grading rubric, dev/test separation.
- Elicitation envelope: explicit named tier (Minimal, Realistic user, Expert scaffold, Threat-model best effort) with the threat model named for Envelope 4.
- Prompts and scaffolds: published at the appropriate level of disclosure given dual-use considerations.
- Tool access: tools used, versions, sandboxing.
- Sampling budget: N, temperature, total tokens, retry policy, selection rule.
- Fine-tuning access: whether fine-tuning was permitted, on what data, with what compute.
- Evaluator profile: time budget, expertise, dev-set iteration history.
- Safety-filter state: deployed, relaxed, or removed; if relaxed, by what method.
- Failure taxonomy: classification of failures as spurious, tradeoff, or real.
- Uncertainty: confidence intervals or variance estimates.
- Falsification conditions: what additional elicitation would invalidate a negative claim, or what attribution test would invalidate a positive claim.
This is a maximalist list. Most current capability claims report some subset. The minimum interpretable subset is the envelope tier, the model and task identifiers, the sampling budget, the tool and scaffold access, and the safety-filter state. A claim that omits any of these is conditional in a way that is not stated.
Net position
Capability evaluation without elicitation specification is not a stable measurement. A score is conditional evidence about what one specific configuration achieved on one specific task; it is not a property of the model.
Capability claims should be made in three forms, depending on the envelope tested and the language used:
- Lower bound: "Under envelope E with budget B, the model achieved score S." This is the safe default for any single-envelope evaluation. It does not claim what the model can or cannot do in general.
- Operational claim: "Under deployment-relevant envelope E₂, the model behaves as follows." This is appropriate for product-behavior claims and accessibility-uplift claims, where the deployed envelope is the right object of measurement.
- Approximate upper bound: "Under threat-model envelope E₄ with documented budget and ablations, the elicited capability is at most C." This is appropriate for safety claims about a specific threat model, with the caveat that no envelope can rule out unknown future elicitation.
"The model cannot do X" without qualification is rarely interpretable. "The model can do X" without qualification is usually attribution drift from a scaffolded system result. The honest summary is the envelope and the score.
The literature has converged on the qualitative claim that elicitation matters and on the necessity of envelope reporting. It has not converged on a quantitative universal multiplier, on a settled operational best-effort standard, or on a robust deception-aware methodology. Confidence is high that the elicitation framing is the right methodological correction; moderate that multi-envelope reporting is the best near-term operational standard; and explicitly low that any current evaluation can be trusted as a robust upper bound on frontier model capability.
Companion entries
Core theory: - Capability Evaluation - Latent Capability - Elicitation Envelope - Threat Modeling for AI Evaluation
Practice: - Scaffolding - Tool Use - Agent Evaluation - Pass@k and Sampling Budgets - Post-Training Enhancement - Red Teaming
Safety evaluation: - AI Safety Evaluation - Responsible Scaling Policies - Model Organisms - Password-Locked Models - Sandbagging and Evaluation Awareness - Dual-Use Research - Responsible Disclosure
Counterarguments and limits: - Benchmark Overfitting - Over-Elicitation and Attribution Drift - Best-Effort Elicitation as a Virtue Word
Adjacent infrastructure: - Context Window Economics - Multi-Model Agent Orchestration Patterns