Production LLM Evaluation: Offline, Online, and the Hybrid Pattern

The practitioner question "how should we evaluate LLM features in production?" rarely has a single right answer. Offline eval suites are fast and controllable but drift and overfit; online experiments are closer to product truth but slow, noisy, and high-stakes; the now-conventional hybrid pattern combines both but can launder weak measurement into the appearance of discipline. This article maps the available evidence onto a purpose-driven operating model: offline evals as regression gates, online signals for product truth and drift, LLM-as-judge as a scalable but bias-prone instrument that must be calibrated against humans, and a feedback architecture that turns production failures into new tests without contaminating held-out measurement.

Coverage note: verified through May 2026.

The practitioner problem

Production LLM features fail in ways that are hard to label, hard to reproduce, and expensive to test directly on users. A retrieval-augmented assistant returns a plausible-sounding but wrong citation. An agent loop completes the task with the wrong tool. A coding model regresses on long-context refactors after a prompt tweak that was supposed to improve formatting. A safety classifier accepts a paraphrased jailbreak that the previous version refused. None of these failures are easy to catch in a single benchmark, and most of them are only fully visible in production traffic, where catching them late is the worst time to catch them.

The discipline of evaluation is the response: a set of measurement instruments meant to convert open-ended model behavior into decisions about whether to ship, tune, monitor, or roll back. Three patterns dominate practitioner discussion:

Offline evaluation: scoring model outputs against curated datasets, deterministic checks, human raters, or LLM judges, in a controlled environment that can run in CI.
Online evaluation: instrumenting production traffic with telemetry, A/B tests, and user-signal analysis to measure what the model actually does for users.
Hybrid evaluation: using offline suites as regression gates and online signals as drift and tuning instruments, with production failures fed back into the offline corpus under explicit governance.

The temptation to treat hybrid as the obvious answer should be resisted long enough to ask what each instrument measures, where each fails, and which decisions each is allowed to support. The taxonomy is useful, but a more important taxonomy lives underneath it: what is the evaluation for? Launch gating, regression detection, prompt or model selection, safety review, drift monitoring, cost and latency control, and product learning are different instruments. A single eval suite cannot honestly serve all of them without becoming vague. See Eval-Driven Development for the lifecycle framing this article uses throughout.

Offline evaluation: speed and its drift

Offline evaluation is the most tractable signal class. A team curates a dataset of inputs and reference outputs (or rubrics for graded outputs), runs candidate systems against it, and produces aggregate scores that can be tracked over time and gated on in CI. Vendor guidance has converged on this shape. OpenAI's evaluation best practices recommend defining objectives, collecting datasets from synthetic, historical, production, and human-curated sources, running evals, and continuously expanding the set (OpenAI Developers). Anthropic's documentation distinguishes code-based grading, human grading, and LLM-based grading, framing code grading as fastest and most reliable where it applies, and LLM grading as scalable but requiring reliability checks (Anthropic).

The strengths are real:

Speed and repeatability. A well-instrumented offline suite produces comparable scores across runs in minutes. This makes it cheap to compare prompts, models, retrieval settings, and decoding configurations.
Regression control. Once a failure is encoded as a test case, future changes can be checked against it before deployment.
Controlled comparison. Holding inputs constant lets engineers attribute changes to system modifications rather than to traffic mix or randomness.

The weaknesses are equally real, and they sharpen as offline suites take on more decision weight.

Drift, overfitting, and the contract problem

Offline suites can quietly become unrepresentative. The dataset reflects whatever failures the team noticed, was willing to label, and was politically free to encode. Real production traffic accumulates new edge cases — new user populations, new product surfaces, new model behaviors — that the suite never sees. Sculley et al. described an analogous failure for traditional ML in Hidden Technical Debt in Machine Learning Systems: data dependencies, feedback loops, and changing external worlds slowly invalidate the assumptions baked into a frozen test set. D'Amour et al.'s work on underspecification (D'Amour et al. 2020) sharpens a related point: aggregate offline metrics often fail to constrain the deployed behavior, because many models with equivalent test scores generalize differently in production.

The second pathology is overfitting in the social sense. Once a suite gates releases, engineers learn its shape. Prompts get phrased so the rubric matches. Few-shot examples drift toward suite distribution. Edge cases are encoded once and never revisited. The suite becomes less a measurement instrument and more a contract: pass these examples, in this rubric, against this judge, under this prompting convention. This is useful for regression prevention and dangerous if mistaken for capability measurement. See Holdout Discipline and Test-Set Contamination for the operational countermeasures.

What offline evaluation is good for

Offline evaluation works best when its decisions match its evidence. It is appropriate as:

A regression gate in CI, blocking obvious breakage before deployment.
A comparison instrument for prompts, retrieval settings, decoding parameters, and model upgrades, so long as comparisons are within the dataset's domain.
A safety guardrail when paired with deterministic checks (refusal patterns, toxicity classifiers, schema validation, tool-use protocols).
A discovery surface for new failure cases when production traces are sampled and labeled.

It is poorly suited for any decision that turns on user-visible product quality, because it cannot, by construction, measure what users do.

Online evaluation: product truth and its costs

Online evaluation instruments production traffic. The signal classes include:

Telemetry: latency, token cost, error rates, tool-call success, retry counts.
User-explicit signals: thumbs up/down, regenerate clicks, accept/reject on completions, escalation to a human agent.
User-implicit signals: session length, return rate, task completion proxies, downstream conversion.
Randomized experiments: A/B tests, multi-armed bandits, shadow deployments, canaries.
Incident review: support tickets, abuse reports, postmortems, manual audits.

Among these, randomized experiments have the strongest causal pedigree. Microsoft's experimentation literature documents A/B testing as a core product-learning infrastructure while emphasizing technical and cultural pitfalls (Kohavi et al., Microsoft Research). Google's overlapping-experiment infrastructure paper describes how high-velocity experimentation is engineered at scale (Tang et al. 2010). For LLM features, this lineage is necessary but not sufficient: experimental rigor is required because intuition is unreliable; sufficiency is harder, because LLM features create measurement problems that classical product experimentation does not have to solve.

Why "online truth" is partially a marketing phrase

The intuition that online metrics are closer to truth than offline metrics is partially correct and partially misleading. Online metrics tell you what users did. They do not automatically tell you whether the answer was correct, safe, useful, or better.

Several confounders are persistent:

Power. A/B tests for LLM features often suffer from small effect sizes, sparse events (retention, escalation), heterogeneous user intent, and limited traffic on enterprise products. Underpowered tests reliably fail to detect regressions that would be obvious to a careful human auditor. See Power Analysis for LLM A/B Tests.
Proxy drift. Thumbs-up rates, session length, and task-completion proxies can reward fluency, polish, persuasiveness, or engagement rather than correctness. A more verbose model that hedges more confidently can win on satisfaction while losing on accuracy.
Novelty effects. New surfaces and new behaviors get clicked more for a few weeks for reasons unrelated to quality.
Mix shifts. Traffic composition changes — geography, device, prompt-template version, upstream router — can move metrics independently of the model under test.
Self-fulfilling telemetry. UI placement, default suggestions, and feedback prompts shape what users do; users adapt to the model's behaviors; the model's outputs then look "successful" because users learned to ask the questions it answers well.
Rare harm. A/B tests cannot statistically detect failures whose base rate is below the test's sensitivity. Many user-visible LLM harms — hallucinated citations, leaked private data, agentic mistakes that escalate — live in this zone.

The combination means online evaluation is essential but not authoritative on its own. It is the best instrument for product-truth questions ("does this change improve or hurt the user-visible product?") and the worst instrument for fine-grained quality questions when traffic is thin or harms are rare.

What online evaluation is good for

Confirming that a candidate system improves user-visible product behavior at a scale that matters.
Detecting drift in latency, cost, refusal rate, tool-success rate, or escalation rate after a deployment.
Catching incidents that no offline test could have anticipated.
Validating segment-level effects (new users, enterprise tier, language) where offline data is sparse.

It is poorly suited for the rapid iteration loop of prompt engineering or for prerelease blocking, because both require speed and statistical power that production traffic rarely provides on the timescale of an engineering sprint.

The hybrid pattern as a control system

The hybrid pattern is the obvious response to the asymmetry: offline gates for fast regression control and comparison; online signals for product truth and drift; production failures continuously converted into new offline tests so the regression suite stays alive. Most major eval-tool vendors describe variants of this loop. LangSmith documents offline evaluation types alongside online monitoring and production feedback loops (LangChain). Braintrust frames its product around experiments, CI evals, online scoring, and feeding production traces back into datasets (Braintrust). Helicone positions itself as observability infrastructure that ingests eval scores rather than as an eval framework on its own (Helicone). Confident AI's DeepEval documents local or remote evals, datasets, tracing, production monitoring, and LLM-as-judge metrics (Confident AI). The convergence is strong evidence of how the practitioner market thinks about the problem; it is weak evidence that any specific stack improves outcomes, because vendors do not run controlled experiments comparing themselves against alternatives.

Hybrid is a pattern, not a conclusion

The risk of treating hybrid as a settled best practice is that it lets organizations feel disciplined without forcing decisions about what "good" means. Offline suites give CI integration and cheap comparisons. Online metrics give apparent realism. LLM judges give scale. Put them together and the team can show a complete loop: traces become datasets, datasets become gates, gates feed releases, releases produce telemetry, telemetry updates traces.

That loop can be self-sealing. Traces overrepresent current users and current failure-discovery mechanisms. Curated datasets overrepresent failures that were easy to notice, easy to label, and politically acceptable to encode. Judges overrepresent the rubric, style, and model-family preferences embedded in their prompt and weights. Online metrics overrepresent measurable behavior. When all four signals agree, the team may have triangulated truth — or it may have built a closed measurement culture around shared blind spots. See Closed-Loop Eval Contamination for the systemic version of this failure.

The defensible version of the hybrid pattern names invalidation conditions explicitly. A hybrid system is working if and only if:

Offline rankings predict online or shadow-mode rankings on at least the high-value comparisons the team makes.
Judge scores correlate with human review at a level the team has measured, not assumed.
Online tests have enough power to detect plausible regressions; underpowered tests are reported as "no measurable change at current N," not as "no change."
Production traces are sampled into discovery sets, and a strict subset is fenced off as a contamination-resistant holdout that does not feed into the iteration loop.
Postmortems for production incidents reliably produce new test cases.

If any of these conditions fail and remain unaddressed, the hybrid pattern is delivering assurance theater rather than measurement. This is not an indictment of the pattern; it is a description of what makes it work.

Lifecycle mapping

A cleaner way to think about hybrid evaluation is by lifecycle stage and decision type:

Stage	Decision	Primary signal	Supporting signal
Development	Will this prompt or model change behavior in the wrong direction?	Offline suite + LLM judge with calibration	Engineer eyeballing of trace samples
CI gate	Does this PR break a known case?	Deterministic checks + curated regression set	Judge score on graded subset
Pre-launch review	Is this candidate ready for traffic?	Adversarial / challenge sets + human review on hard slices	Calibration check on judge
Canary / shadow	Does this candidate match or beat the incumbent on real traffic?	Shadow scoring + telemetry	Judge on sampled traces
A/B test	Does this candidate improve user-visible product?	Randomized experiment with power analysis	Guardrail metrics, segment analysis
Production monitoring	Has anything degraded since launch?	Telemetry + online judge sampling + escalation rate	Drift in trace distribution
Eval refresh	Does the offline suite still represent production?	Sampled trace coverage analysis	Incident-driven test additions

Each stage has its own dominant signal and its own characteristic failure mode. Conflating stages is a major source of evaluation pathology. A judge useful for development is not necessarily strong enough for compliance gating. An A/B test that decides product direction is not the right instrument for blocking a CI commit. The hybrid pattern works when each instrument is used for its calibrated purpose and not allowed to substitute for another.

A useful structural disagreement

There is a real practitioner disagreement about whether the hybrid pattern should be the article's recommendation at all. The skeptical objection is that "offline plus online plus judge" is a recipe for a complete-looking loop that can quietly drift away from truth on every channel simultaneously, and that the right move is to refuse to recommend hybrid as a default until each channel has been independently validated against ground truth on the team's actual product. The architectural reply is that the alternative — picking one channel and pretending the others do not exist — is worse: every production team that ships LLM features ends up with multiple signal channels eventually, and an explicit hybrid architecture with named invalidation conditions is the honest way to admit that.

The right resolution is conditional. Hybrid is the best-supported default given invalidation conditions and calibration. Without those, it is dashboards, not measurement. With them, it is the cleanest pattern available. The disagreement is not about whether to use multiple signals; it is about how much of the discipline can be assumed and how much has to be built.

LLM-as-judge: scaling measurement, importing bias

LLM-as-judge is the technology that makes scalable offline evaluation feasible for open-ended outputs. Rather than writing brittle string-match rubrics or paying for human labels at every iteration, the team prompts a strong model to grade outputs against a rubric. The empirical case for this is real: in Zheng et al. 2023, LLM judges (notably GPT-4-class models) reached agreement rates with human labelers that matched human-human agreement on chatbot preference benchmarks, often above 80%. This is the foundation on which most current eval tooling rests.

The empirical case against treating judge models as ground truth is also real, and it has gotten stronger over time.

Documented biases

The biases are not incidental. They are structural properties of how judge models are prompted and how they generate.

Position bias. Judges systematically prefer one position (often the first, sometimes the last) when comparing two candidates. The bias varies by judge model, task, and the gap in candidate quality. Shi et al. (2024) formalized this and showed that swapping order can flip a non-trivial fraction of judgments. See Position Bias in LLM Judges.
Verbosity bias. Judges prefer longer, more polished, more confidently structured outputs even when the shorter output is correct. This is the failure mode that most directly poisons rubrics for "helpfulness" or "quality" without explicit length controls. See Verbosity Bias in LLM Judges.
Self-preference / familiarity bias. Judges score outputs from their own model family more favorably. Wataoka et al. (2024) measured this directly across multiple judge-candidate combinations. See Self-Preference Bias in LLM Judges.
Limited reasoning. Judges struggle on tasks requiring genuine domain expertise, multi-step reasoning, or fact-checking against external sources. The original MT-Bench paper named this; later work has mostly reinforced it.
Rubric drift. Subtle changes to the judge prompt move scores by amounts comparable to the differences between candidate systems. This makes year-over-year score comparisons fragile when rubrics evolve.

The same papers that motivate using LLM judges document why they cannot be used naively. OpenAI's eval guidance reflects this: model-graded checks are listed alongside string checks, similarity checks, and code-execution graders rather than presented as a superior class (OpenAI Developers). Anthropic's docs frame LLM grading as scalable but requiring reliability checks (Anthropic).

Calibration is the price of using judges

Treating LLM-as-judge as a measurement instrument rather than as ground truth means a calibration step. The minimum viable calibration includes:

Human-labeled gold set. A representative-and-adversarial sample of inputs, labeled by domain-qualified humans against the same rubric the judge will use. This is the bedrock; everything else floats on its quality.
Agreement measurement. Compute judge-human agreement with appropriate metrics (accuracy, Cohen's κ, Krippendorff's α for graded scales). Report agreement per slice; aggregate agreement hides per-slice failures.
Bias audits. Run candidates in shuffled order, with verbosity controls, with blind labels, and with multiple judge models. Quantify position sensitivity, verbosity sensitivity, and self-preference.
Re-calibration on judge upgrade. A new judge model is a new measurement instrument. Re-run the audit before letting it gate releases.
Per-domain validity. GPT-4-class agreement on chat preference benchmarks does not establish validity for legal advice, medical triage, RAG faithfulness, agentic tool use, or safety-classifier correctness. Each new domain needs its own calibration. See Calibration Studies for Judge Models.

A shorthand: if a team cannot say what their judge-human agreement is on the slices that matter, the judge is not a measurement instrument. It is a vibe.

Where judges legitimately scale evaluation

After calibration, judges are useful for:

Scoring large rubric-based suites where deterministic checks are insufficient.
Triaging large trace datasets so human reviewers see the most diagnostic samples.
Online sampled scoring at production scale, with the understanding that drift in judge agreement is itself a signal.
Cross-checking other judges (rotating judge models per evaluation cycle) to detect family-level bias.

Judges do not substitute for human review on high-stakes decisions, do not establish ground truth, and do not produce comparable numbers across rubric or judge changes without re-calibration. The case against using judges at all in safety-critical or compliance-sensitive paths is plausible and worth taking seriously; see The Case Against LLM-as-Judge.

Failure modes

Each evaluation pattern has characteristic ways of going wrong. The failure-mode matrix is the part of this article that practitioners are most likely to need on the job.

Offline-eval drift and dataset rot

Symptoms: offline scores improve while support tickets, human review failures, or online guardrail events rise. The suite is no longer measuring what users encounter.

Causes: production traffic distribution has moved; new failure modes have emerged that the suite does not cover; existing tests have been overfit through repeated tuning.

Mitigations: refresh cadence (with a target percentage of test cases drawn from recent production samples), adversarial example generation, contamination-resistant holdouts, slice-level reporting (not just aggregate), and explicit retirement of stale tests rather than indefinite accretion.

Detection signals: divergence between offline trends and online trends; lack of any new tests in the past N months; aggregate scores that mask per-slice regressions.

Test-set leakage

Symptoms: offline scores are uniformly high; human review reveals trivial errors that the rubric should have caught.

Causes: the test set leaked into training (for fine-tuned models), into the prompt (via few-shot examples), or into the judge's prompt (via implicit hints). For RAG systems, the retrieval index can include the test corpus.

Mitigations: separate development sets from gating sets from frozen holdouts; never use the gating set's content in prompts or training data; for RAG, partition by document or by user when possible.

Detection signals: anomalously high scores on a new system; near-perfect performance on a small subset; sensitivity to minor prompt changes that should not matter.

Judge bias

Symptoms: judge scores shift sharply after a judge-model upgrade, even though the candidate systems did not change. Or: rankings reverse when candidate order is swapped. Or: a verbose system wins on the judge and loses on human review.

Causes: position, verbosity, self-preference, and rubric-interpretation biases as documented above.

Mitigations: order shuffling, verbosity controls, multi-judge committees, judge rotation, human-audited samples, blind labels.

Detection signals: large judge-score swings after upgrades with no candidate change; position sensitivity above the rubric's tolerance; growing judge-human disagreement on calibration sets.

Underpowered A/B tests

Symptoms: tests "fail to find" effects that human review or offline analysis suggest are real. Decisions get made on directional movements within the noise floor.

Causes: low traffic, sparse outcome events, high variance in the metric, multiple-comparisons inflation, peeking and early stopping.

Mitigations: pre-registered power analysis; sequential testing methods with corrected stopping rules; A/A tests to estimate baseline noise; minimum detectable effect (MDE) reporting; willingness to say "we cannot detect this with available traffic" and rely on offline plus human review for the decision.

Detection signals: extremely wide confidence intervals; metric movements only in aggregate; contradictory satisfaction and task-success signals.

Telemetry confounders

Symptoms: metrics move after a deployment in ways that do not reproduce on rollback; segment-level effects contradict aggregate effects; satisfaction rises while correctness audits show degradation.

Causes: novelty effects, traffic mix shifts, UI changes coincident with model changes, upstream router changes, seasonal demand, incident response artifacts.

Mitigations: holdout populations, guardrail metrics, segment analysis, change-management discipline (one change at a time when possible), waiting through novelty windows before declaring victory.

Detection signals: metric movements that fail to replicate; effects that vanish in segment views; correlations with calendar events.

Privacy and retention exposure

Symptoms: prompts, completions, user identifiers, or sensitive business data appear in eval traces, dashboards, or exported datasets.

Causes: trace logging without minimization; broad dashboard access; eval exports that leave controlled systems; insufficient redaction; long retention.

Mitigations: minimization at logging time, redaction pipelines, retention limits, access controls, differential sampling for high-sensitivity surfaces, legal and security review for telemetry that includes user content.

Detection signals: sensitive fields appearing in traces; inability to answer "who can read this dashboard?"; eval data crossing tenant boundaries.

Closed-loop eval contamination

Symptoms: offline scores keep going up; online behavior does not improve at the same rate; the team starts trusting the offline number more than the production reality.

Causes: production traces feed back into the same suite that gates releases without separation between discovery and gating; engineers tune to the suite; the suite stops representing the population it was meant to.

Mitigations: explicit separation between discovery sets, regression gates, challenge sets, and frozen holdouts; rotation of holdouts on a schedule; periodic external review of the suite's coverage.

Detection signals: offline-online divergence widening over time; stagnation in human-review pass rates despite rising offline scores; new incident classes that no test would have caught.

Organizational metric gaming

Symptoms: teams optimize for the eval, not the user. Reported wins on judge scores or A/B metrics do not produce the user-visible improvements stakeholders expected.

Causes: eval scores tied to performance reviews, OKRs, or release authority; lack of accountability for postmortem-driven test additions; lack of independent audit.

Mitigations: separate the eval team from the team being evaluated when stakes are high; make the audit function explicitly responsible for invalidation rather than for cheerleading; rotate metric definitions periodically; require human-review evidence alongside automated scores for major launches.

Detection signals: improvements that do not generalize; a culture in which questioning the eval is treated as obstruction; suspiciously smooth quarter-over-quarter score growth.

The tooling landscape

The tooling market for production LLM evaluation has converged on a recognizable set of capabilities. Mapping tools by capability rather than by vendor is more durable than a feature comparison that goes stale within a quarter.

Capability classes

Capability	What it does	Examples
Tracing and observability	Captures inputs, outputs, intermediate tool calls, latency, cost per request	Helicone, LangSmith, Braintrust, Confident AI, in-house OpenTelemetry pipelines
Dataset management	Versioned eval datasets, sampling from traces, annotation, holdout discipline	LangSmith, Braintrust, Confident AI, custom dataset stores
Eval execution	Running candidates against datasets, computing scores, reporting deltas	LangSmith, Braintrust, OpenAI Evals/Graders, DeepEval, in-house harnesses
Human annotation	Queueing samples for human review, capturing rubric-graded labels	LangSmith, Braintrust, Confident AI, Argilla, in-house annotation tools
LLM-as-judge orchestration	Prompting judge models, handling order shuffling, multi-judge committees	DeepEval, Braintrust, LangSmith, OpenAI Graders, custom judge harnesses
CI integration	Running evals on pull requests, gating merges, posting deltas	Braintrust GitHub Actions, LangSmith CI, custom CI scripts
Online scoring and monitoring	Sampled judge scoring on production traffic, drift dashboards, alerting	Helicone, LangSmith, Braintrust, Datadog/Grafana with custom metrics
Experiment analysis	Power analysis, segment breakdowns, sequential testing, guardrails	Statsig, Eppo, Optimizely, in-house experimentation platforms
Governance	Access control, dataset versioning, audit logs, retention policies	All major platforms to varying degrees; in-house tooling for regulated settings

What vendor sources can and cannot tell you

OpenAI, Anthropic, LangSmith, Braintrust, Helicone, and Confident AI are primary sources for what their tools support. Their docs reliably tell you what features exist, how the APIs are shaped, and what workflows the vendors expect. They are weak evidence that a particular tool improves production outcomes, because vendors publish capability claims rather than controlled studies.

Practitioners reading vendor blogs should distinguish three claims:

The mechanism exists (the tool supports judge-model orchestration with order shuffling).
The mechanism works (calibrated judges produce stable rankings on the team's tasks).
The mechanism works in this product class (judge stability has been validated for the team's specific domain, traffic, and decision types).

Vendor sources are sufficient evidence for (1) and weak evidence for (2) and (3). Independent papers, postmortems, and product-local validation are the evidence that carries (2) and (3). Articles that conflate these levels turn into market maps dressed up as engineering guidance. See Vendor Evidence and Source Quality in AI Engineering.

The build-vs-buy question

In-house systems remain common, especially at organizations with strong existing experimentation infrastructure (where adding LLM-specific instrumentation to an existing platform is cheaper than adopting a new vendor). The build-vs-buy decision turns on:

Schema control. In-house systems can encode domain-specific trace schemas; vendors require working within their data model.
Privacy and tenancy. Vendor-hosted eval data may be unacceptable for regulated industries or for products handling sensitive user content.
Integration with existing experimentation. Teams with mature A/B platforms often prefer to extend rather than fork.
Velocity of feature change. Vendor platforms ship features faster than most in-house teams; this advantage decays as in-house teams catch up.
Lock-in cost. Trace data and dataset format are sticky; migration is expensive after a few quarters of usage.

There is no universal answer. The most common pattern is using a vendor for tracing and dataset management early, with in-house systems gradually absorbing the workflow as the team's evaluation needs become product-specific. See Build vs Buy for LLM Eval Infrastructure.

Is production LLM eval new, or a continuation?

The active practitioner debate is whether LLM evaluation is fundamentally different from production ML evaluation or a continuation of the same discipline. Both extreme framings are wrong. The honest answer is that it is mostly continuation at the control-system level and materially different at the measurement layer.

Continuity

The control-system patterns transfer cleanly. Train/test separation, holdout discipline, drift monitoring, online experimentation, telemetry instrumentation, and feedback-loop management are the same problems they have always been. The pathologies named in Hidden Technical Debt in Machine Learning Systems (Sculley et al. 2015) — data dependencies, undeclared consumers, glue code, configuration debt, feedback loops — show up unchanged in LLM systems. The underspecification problem from D'Amour et al. shows up unchanged: many models with equivalent offline scores generalize differently in production.

The experimentation discipline transfers cleanly too. Power analysis, guardrail metrics, A/A tests, sequential stopping rules, segment analysis, and multiple-comparison correction are the same statistics they were in classical product experimentation (Microsoft Research, Tang et al.). LLM features do not require a new statistics; they require honest application of the existing statistics, often under conditions of low traffic and rare events that make the statistics harder to apply well.

Discontinuity

What is new is concentrated in the measurement layer:

Open-ended outputs. Traditional ML evaluation often has a label space (classification, ranking, regression). LLM outputs are arbitrary text or trajectories. Reducing them to a number requires a rubric, and the rubric is a piece of product code that must be versioned and audited.
Prompts as product code. A prompt change is a system change. Without instrumentation that treats prompts as versioned artifacts, there is no way to reason about which version produced which behavior. See Prompt Regression Testing.
Model churn. Foundation models change underneath the application. A vendor model upgrade can move every metric without any team-side change. Classical ML systems mostly did not have this property at the timescales LLM teams now experience.
Models as measurement instruments. Using a model to grade another model is a category that classical ML evaluation did not need. It introduces a new failure mode: the measurement instrument has its own biases, drifts on its own upgrade schedule, and may have correlated failure modes with the system being measured.
Tool trajectories. Agentic systems have execution traces, not just outputs. A correct final answer can hide an unsafe tool call. Classical ML evaluation did not have a comparable structure.
Weak ground truth. For many LLM tasks, "correct" is contested even among experts. Inter-annotator agreement on subjective rubrics is often lower than the differences between candidate systems. This is not a property of LLMs per se; it is a property of the open-ended tasks that LLMs make tractable, and it forces evaluation methodology to do work that previous ML evaluation could often skip. See Inter-Annotator Agreement and Weak Ground Truth.

The right framing is that production LLM evaluation is production ML evaluation with worse observability, weaker labels, and a new measurement instrument that is itself a model. Practitioners are not abandoning the ML control system; they are extending it under harder conditions. Articles that frame LLM eval as a wholly new discipline tend to underrate the maturity of the underlying patterns. Articles that frame it as "just ML eval" tend to underrate how much new instrumentation is required to make those patterns work. See Is LLM Eval Fundamentally New? for the full version of this debate.

A decisive experiment for any production team

A team that wants to know whether its hybrid eval system is working — rather than performing — can run a comparatively cheap calibration study. The shape, with rough sample sizes:

Sample 200–500 production or beta traces across slices: normal usage, edge cases, known failure modes, and adversarial probes. Stratify so the sample is representative-and-adversarial, not just representative.
Reserve a contamination-resistant holdout (≥30% of the sample) that does not enter the iteration loop. This subset will be used to evaluate whether changes to the eval system itself improve or degrade alignment with truth.
Have domain-qualified humans label 100–200 examples against the same rubric the judge will use, with explicit grading criteria and a disagreement-resolution protocol.
Run candidate systems (current production, proposed change) and at least two judge models with order shuffling, verbosity controls, and blind labels. Record per-judge, per-position, per-candidate scores.
Compute judge–human agreement overall and by slice. Compute position sensitivity and verbosity sensitivity. Identify slices where judge-human agreement falls below the team's tolerance.
Run an offline ranking of the candidates on the calibrated judge.
Run a powered canary, shadow deployment, or A/B test at a sample size determined by a pre-registered power analysis with explicit minimum detectable effect.
Compare offline-judge ranking with online ranking. If they agree on the high-value comparisons, the offline system has predictive validity for those comparisons. If they disagree, the offline system is not yet a release gate, and the team should know so.
Repeat on a cadence — per major model upgrade, per quarter, per significant rubric change.

The point of the experiment is not to produce a single number. It is to expose, in a way the team cannot ignore, whether their offline system actually predicts the production decisions it is being used to make. If it does, the hybrid pattern is working as designed. If it does not, the team has discovered the most important thing about its evaluation system: that the system is not yet a control system, and that decisions made on its outputs are, for now, decisions made on instinct dressed up as data.

What to take from this

The hybrid pattern is the best-supported operating model for production LLM evaluation, but the strength of "best-supported" is moderate, not high. Public evidence is dominated by vendor docs (primary for capabilities, weak for outcomes), product engineering blogs (helpful for patterns, selective in what they publish), and a research literature that is strong on judge bias and underspecification but thin on cross-product causal evidence that any specific eval stack improves user outcomes. The prudent operating posture is:

Use offline evaluation as a regression and comparison instrument, with explicit holdout discipline and refresh cadence.
Use online evaluation for product-truth decisions, with explicit power analysis and willingness to report "no measurable change at current N."
Use LLM-as-judge as a scalable measurement aid that has been calibrated against human labels on the slices that matter, with bias audits on every judge upgrade.
Treat the hybrid loop as a control system with named invalidation conditions, not as a maturity stage to graduate into.
Treat tooling as implementation: choose by capability fit and governance fit, not by feature checklist.
Treat the "is this new ML or new science?" question as a continuity story with a hard new measurement layer; resist marketing language in either direction.

The strongest claim the available evidence supports is that production LLM evaluation requires more explicit measurement governance than classical ML, not less. The discipline is still the discipline; the conditions are harder, and the temptations to substitute dashboards for measurement are stronger. Teams that name what would prove their eval system wrong, and check, will end up with eval systems that work. Teams that build elaborate hybrid loops without invalidation conditions will end up with elaborate hybrid loops.

Companion entries

Core theory: - LLM-as-Judge - Evaluation Drift - Underspecification in ML Systems - Hidden Technical Debt in ML - Inter-Annotator Agreement and Weak Ground Truth - Holdout Discipline and Test-Set Contamination

Practice: - Prompt Regression Testing - Trace-Derived Eval Sets - Adversarial Eval Construction - Calibration Studies for Judge Models - Power Analysis for LLM A/B Tests - Shadow Deployments and Canary Evaluation - Production ML Monitoring - Eval-Driven Development

Tooling: - Tracing and Observability for LLM Applications - Dataset Management for Evals - CI Gating for Prompt and Model Changes - Online Scoring at Production Scale - Human Annotation Workflows - Build vs Buy for LLM Eval Infrastructure

Failure modes: - Position Bias in LLM Judges - Verbosity Bias in LLM Judges - Self-Preference Bias in LLM Judges - Underpowered Online Experiments - Telemetry Confounding - Closed-Loop Eval Contamination - Privacy and Retention Risk in Eval Pipelines - Organizational Metric Gaming

Counterarguments: - Offline-First Evaluation for Narrow Domains - Online-First Evaluation for High-Traffic Products - The Case Against LLM-as-Judge - Is LLM Eval Fundamentally New? - Vendor Evidence and Source Quality in AI Engineering