Item Response Theory (IRT) reframes assessment as a probabilistic model: the probability of a response is a function of a person’s latent ability and an item’s statistical properties. This shift from Classical Test Theory to Latent Variable Models makes calibrated item banks, score equating, computerized adaptive testing, and targeted measurement precision possible, but only when strong assumptions about dimensionality, item fit, and population invariance are credible. For AI evaluation, IRT offers a disciplined language for benchmark construction and judge calibration, while also exposing an unresolved problem: LLM “ability” and item difficulty may not be invariant in the way human psychometric applications require.

Coverage note: verified through May 19, 2026.

Item Response Theory: Modeling Probability of Response as a Function of Ability

1. The foundational shift: from observed scores to response models

Classical test theory, or Classical Test Theory, treats an observed test score as a fallible measurement of a true score. In its familiar form, an examinee’s observed score is decomposed into true score plus error; reliability is then an observed-score property of a test in a population, often summarized as the ratio of true-score variance to observed-score variance. Lord and Novick’s Statistical Theories of Mental Test Scores became the canonical mid-century synthesis of this tradition and its extensions; the 1968 book was published by Addison-Wesley with contributions by Allan Birnbaum. ([Lord & Novick 1968 review]) Sage Journals

The limitation was not that sum scores were useless. Sum scores remain practical, interpretable, and often highly correlated with IRT ability estimates. The limitation was that classical summaries attach properties to whole tests and observed samples: an item’s “difficulty” is usually the proportion of respondents who answer it correctly in a given group, and an item’s discrimination is usually its relationship to a total score in that same group. Change the group or the form, and the classical item statistics change.

IRT makes a different modeling move. Instead of beginning with the total score, it begins with the item response. The primitive object is not “What total score did this person obtain?” but “What is the probability that person p responds correctly, endorses, or receives category k on item i, given a latent trait value?” Lord’s later Applications of Item Response Theory to Practical Testing Problems explicitly presented IRT as a practical framework for item analysis, ability estimates as functions of item parameters, information functions, tailored testing, mastery testing, equating, item-bias study, and related testing problems. ([Lord 1980, ERIC record]) ERIC

Rasch’s 1960 Probabilistic Models for Some Intelligence and Attainment Tests supplied the foundational one-parameter model in which the probability of success is governed by the difference between person ability and item difficulty. Rasch’s contribution is often grouped with the “1PL” model, though Rasch measurement has a distinct philosophical emphasis on invariant comparison and model-fit discipline rather than merely choosing a parsimonious logistic regression for item responses. ([Rasch 1960 review record]) Cambridge University Press & Assessment

Dimension	Classical test theory	Item response theory
Primitive object	Observed test score, usually a sum or scale score	Item-level response probability
Person score	Total score or transformed total score	Latent trait estimate, usually θ\thetaθ
Item difficulty	Sample proportion correct or mean endorsement	Location parameter on latent trait scale
Item discrimination	Correlation with total score or related index	Slope parameter in the response function
Reliability	Often a single test-level coefficient for a population	Conditional precision as a function of θ\thetaθ, via information
Comparability	Achieved through forms, equating, and reliability studies	Achieved through calibrated item parameters, linking, and model fit
Main weakness	Item and score statistics are tied to sample and form	Strong assumptions; misfit can make parameter estimates misleading

The central IRT claim is not that sum scores are “wrong.” It is that sum scores throw away information that becomes crucial when items differ in difficulty, discrimination, guessing susceptibility, or category thresholds. IRT replaces the test-as-bag-of-equivalent-items assumption with a model in which item and person parameters are estimated on a common latent scale.

2. The mathematical core: probability as a function of ability and item parameters

For a dichotomous item, the canonical three-parameter logistic model is:

P(Xpi=1∣θp)=ci+(1−ci)11+exp⁡[−Dai(θp−bi)]P(X_{pi}=1 \mid \theta_p)

c_i + (1-c_i)\frac{1}{1+\exp[-D a_i(\theta_p-b_i)]}P(Xpi=1∣θp)=ci+(1−ci)1+exp[−Dai(θp−bi)]1 where:

Symbol	Meaning	Interpretation
θp\theta_pθp	Person ability or trait level	The latent position of person ppp on the measured construct
bib_ibi	Item difficulty or location	The point on the latent scale where the item is centered
aia_iai	Item discrimination	The steepness of the item characteristic curve
cic_ici	Pseudo-guessing parameter	Lower asymptote: probability of a correct response for very low θ\thetaθ
DDD	Scaling constant	Often 1, sometimes 1.7 to approximate the normal-ogive metric

The curve defined by this function is the item characteristic curve, or Item Characteristic Curve. If aia_iai is high, small changes in θ\thetaθ near the item’s location produce large changes in response probability; the item is informative for distinguishing nearby persons. If aia_iai is low, the item gives weak evidence about ability. If bib_ibi is high, only persons high on the latent trait have a substantial probability of success. If cic_ici is high, even very low-θ\thetaθ persons have a nontrivial chance of a correct answer, usually because of guessing or artifact.

This is the separability that makes IRT powerful. Persons and items are not collapsed into one total-score table. Items have parameters. Persons have parameters. A response pattern is evidence about both.

In the Rasch model, the dichotomous form is usually written as:

P(Xpi=1∣θp,bi)=exp⁡(θp−bi)1+exp⁡(θp−bi)P(X_{pi}=1 \mid \theta_p, b_i)

\frac{\exp(\theta_p-b_i)}{1+\exp(\theta_p-b_i)}P(Xpi=1∣θp,bi)=1+exp(θp−bi)exp(θp−bi) There is no item-specific discrimination parameter and no guessing parameter. All items are assumed to share the same slope, and the sufficient statistic for person ability is the raw score, conditional on the fitted item difficulties. This makes Rasch models especially attractive when the goal is defensible measurement rather than maximum descriptive fit.

In the 2PL model, aia_iai varies by item. In the 3PL model, cic_ici is added for multiple-choice or otherwise guessable items. In practice, this is a tradeoff: additional parameters may better fit observed data, but they demand larger calibration samples and can reduce the strict invariance properties associated with the Rasch family.

3. The model family

IRT is not one model. It is a family of models for ordered, unordered, dichotomous, polytomous, unidimensional, and multidimensional response data. Embretson and Reise’s Item Response Theory for Psychologists is a standard bridge text for psychological applications; the book surveys contemporary IRT models, estimation methods, software, and gives central attention to polytomous models because many psychological instruments use rating scales rather than right/wrong items. ([Embretson & Reise]) Google Books

Model	Parameters	Response type	Best use	Main constraint
Rasch / strict 1PL	bib_ibi, θp\theta_pθp	Dichotomous	Measurement with strong invariance goals	Equal discrimination, no guessing
1PL with common slope	bib_ibi, common aaa, θp\theta_pθp	Dichotomous	Parsimonious logistic IRT	Item slopes constrained equal
2PL	ai,bi,θpa_i, b_i, \theta_pai,bi,θp	Dichotomous	Items vary in discrimination	Requires more data; raw score no longer sufficient
3PL	ai,bi,ci,θpa_i, b_i, c_i, \theta_pai,bi,ci,θp	Dichotomous, often multiple-choice	Guessing-prone selected-response tests	Harder estimation; cic_ici can be unstable
Graded Response Model	aia_iai, ordered thresholds	Ordered categories	Likert, ratings, graded responses	Assumes ordered cumulative category structure
Partial Credit Model	Step difficulties	Ordered categories	Rubrics, partial-credit scoring	Rasch-family constraints
Generalized Partial Credit Model	Discrimination plus step parameters	Ordered categories	Polytomous data with varying slopes	More flexible, more demanding
Rating Scale Model	Item locations plus common category structure	Ordered categories	Shared rating scale across items	Category thresholds constrained across items
Multidimensional IRT	Multiple θ\thetaθ values	Any supported response type	Multiple latent traits	Harder identification, calibration, interpretation

Samejima’s graded-response work is foundational for ordered polytomous responses: it models latent ability using graded score patterns rather than binary right/wrong outcomes. ([Samejima graded-response monograph record]) Cambridge University Press & Assessment Masters’s Partial Credit Model is a Rasch-family model for responses recorded in two or more ordered categories, designed for the same general class of problems as graded-response models but with a different parameterization and measurement philosophy. ([Partial Credit Model, Springer record]) Springer

For AI evaluation, the polytomous models matter as much as the dichotomous ones. Many AI tasks are not simply correct or incorrect. Rubric-based writing evaluation, code review, tool-use traces, safety assessments, and preference judgments often produce ordered categories: unacceptable, weak, acceptable, strong; or 1–5 ratings; or partial credit across rubric dimensions. Treating those as binary pass/fail outcomes discards information and can create brittle benchmarks.

4. Foundational properties: invariance, information, and separability

IRT’s most important promise is invariance. Under the model, item parameters should be stable across populations, and person parameters should be stable across item subsets. This is the conceptual heart of Measurement Invariance.

There are two forms:

Invariance claim	Meaning	Practical test
Item-parameter invariance	Item difficulty, discrimination, and guessing estimates should not depend materially on which representative population calibrated them	Calibrate items in subgroups; inspect drift, DIF, and linking stability
Person-parameter invariance	A person’s θ\thetaθ estimate should not depend materially on which calibrated subset of suitable items they received	Compare estimates across forms, booklets, panels, or adaptive routes

This does not mean parameters are metaphysical constants. The invariance claim is conditional: it holds only if the model is appropriate, the latent trait is sufficiently coherent, local independence is plausible, items are not functioning differentially across groups, and the calibration is statistically adequate. A Columbia methods summary lists the standard assumptions as monotonicity, unidimensionality, local independence, and invariance, and explicitly notes that item parameters can be estimated from any group only if the assumptions hold. ([Columbia IRT methods overview]) Mailman School of Public Health

The practical interpretation should be severe: IRT provides a way to test and exploit invariance, not a guarantee that invariance exists. If an item is easier for one subgroup for reasons unrelated to the target construct, the item exhibits Differential Item Functioning. If two items share a passage, template, or hidden skill, local independence can fail. If a test mixes algebra, reading comprehension, spatial reasoning, and test-taking speed, a single θ\thetaθ may be an oversimplification.

The second major property is information. In classical reliability, one often asks: “How reliable is this test?” In IRT, the sharper question is: “How much information does this item or form provide at each point on the ability scale?”

For a simple 2PL item without guessing, item information is proportional to:

Ii(θ)∝ai2Pi(θ)(1−Pi(θ))I_i(\theta) \propto a_i^2 P_i(\theta)(1-P_i(\theta))Ii(θ)∝ai2Pi(θ)(1−Pi(θ)) The test information function is the sum of item information:

IT(θ)=∑iIi(θ)I_T(\theta)=\sum_i I_i(\theta)IT(θ)=i∑Ii(θ) The standard error of θ\thetaθ is approximately:

SE(θ)≈1IT(θ)SE(\theta) \approx \frac{1}{\sqrt{I_T(\theta)}}SE(θ)≈IT(θ)1 This is why IRT is not merely a scoring theory. It is a design theory. If an assessment needs high precision near a certification cutoff, the item bank can be built to concentrate information near that cutoff. If an assessment needs high precision across a broad range, the bank needs items distributed across that range. If an adaptive test is estimating a person near θ=1.5\theta=1.5θ=1.5, items with high information near θ=1.5\theta=1.5θ=1.5 are more valuable than items centered at θ=−1.5\theta=-1.5θ=−1.5.

5. Estimation: calibration, scoring, and scale identification

IRT estimates are not absolute. The θ\thetaθ scale must be identified by constraints, usually by setting a reference population mean and standard deviation, or fixing item-parameter constraints. If all θ\thetaθ values and item difficulties are shifted together, response probabilities do not change. If slopes and scale are transformed together, some models preserve equivalent probabilities. This is why linking and scale anchoring matter.

A typical IRT workflow has four stages:

Item calibration. Estimate item parameters from response data. This can use joint maximum likelihood, marginal maximum likelihood, conditional maximum likelihood for Rasch-family models, or Bayesian methods.
Model-fit evaluation. Inspect item fit, person fit, residual dependence, dimensionality, category functioning, parameter drift, and subgroup DIF.
Scale linking. Place new items, forms, or populations on an existing scale through anchor items, common-person designs, common-item designs, or external constraints.
Scoring. Estimate person ability using maximum likelihood, expected a posteriori, maximum a posteriori, plausible values, or other procedures appropriate to the use case.

Large-sample requirements are real. There is no universal NNN, because calibration precision depends on model family, item quality, targeting, missingness, dimensionality, and estimation method. A 2025 tutorial argues for simulation-based sample-size planning in IRT because precise item/person estimates, structural effects, and model-fit results depend on the specific design; it also identifies larger sample-size requirements as one reason IRT is underused in practice. ([Sage tutorial on IRT sample-size planning]) Sage Journals A 2021 simulation paper summarizes older guidance that sample sizes around 500 for 2PL and 1,000 for 3PL were often recommended, with more accurate results around 2,000, but those numbers should be treated as rough historical heuristics rather than portable rules. ([IRT simulation paper via ERIC PDF]) files.eric.ed.gov

For AI evaluation, this point is especially important. A benchmark with 200 prompts and 10 models may be too sparse to support stable 2PL or 3PL item calibration. A benchmark with 20,000 item-model responses may still be poorly identified if all items are near-saturated, if the tasks are heavily templated, or if judge noise dominates item signal.

6. Empirical applications: large-scale assessments

IRT became operationally important because it solved practical testing problems: equating, form construction, item banking, adaptive routing, and precision targeting.

SAT and multistage adaptive testing

The digital SAT Suite is a current example of IRT-based large-scale assessment. College Board’s technical manual describes the digital SAT Suite as a multistage adaptive testing design: students first receive a routing module covering a broad difficulty spectrum, and their performance determines whether their second-stage module is, on average, higher or lower difficulty. The same manual states that the tests are constructed and scored with item parameters estimated from pretesting and that statistical specifications are articulated in terms of IRT test information functions and test characteristic curves. ([College Board Digital SAT Technical Manual]) research.collegeboard.org

The manual is explicit about why information functions matter operationally: target TIFs are set at points on the ability continuum so reliability can be maintained regardless of whether a student routes to the higher- or lower-difficulty module, and automated assembly minimizes the difference between target and estimated TIFs. ([College Board Digital SAT Technical Manual]) research.collegeboard.org

This is not item-by-item computerized adaptive testing. It is Multistage Adaptive Testing, a constrained compromise among psychometric precision, content balance, exposure control, security, and operational simplicity. The point remains: without calibrated item parameters and information functions, this design is not available in the same way.

GRE

ETS’s research compendium for the revised GRE General Test describes the operational need clearly: when testing is delivered frequently or continuously, IRT is generally required to produce comparable scores across many forms. The GRE work considered 3PL and 2PL alternatives; the 3PL accounts for difficulty, discrimination, and guessing, while the 2PL accounts for difficulty and discrimination. ETS reported that the 2PL fit Verbal Reasoning and Quantitative Reasoning tryout data well and selected it for the revised GRE General Test partly because reduced simple multiple-choice usage reduced successful guessing and because the 2PL has fewer parameters to estimate. ([ETS GRE research compendium]) ETS

This is a useful reminder that “more parameters” is not automatically better. In operational testing, model choice is constrained by fit, sample size, score comparability, item types, security, and interpretability.

NAEP

NAEP uses IRT to place students who receive different item blocks onto common score scales. The NAEP technical documentation states that IRT models describe relationships between item responses and underlying score scales, provide a common scale for students receiving different blocks, and estimate item parameters from response data. It also notes that different IRT models are used for multiple-choice, dichotomous constructed-response, and polytomous constructed-response items, with model fit evaluated during item parameter estimation. ([NAEP score-scale documentation]) National Center for Education Statistics

NAEP’s item-parameter documentation states that during scaling, parameters are estimated using current and recent past assessment data, with 2PL items estimating aaa and bbb, 3PL items estimating aaa, bbb, and ccc, and generalized partial-credit items estimating aaa, bbb, and ddd parameters. NAEP also notes that person parameters are estimated during scaling, though group reporting does not rely directly on individual parameter estimates. ([NAEP IRT parameter documentation]) National Center for Education Statistics

For AI evals, NAEP’s matrix-sampling logic is relevant: not every examinee needs every item if the design supports common-scale estimation. That matters for expensive model evaluations, where a full benchmark run can be costly.

7. Modern instrument development and SAPA/IPIP

IRT also changed psychological instrument development. Instead of treating a personality inventory, symptom scale, or attitude questionnaire as a fixed set of items with a single alpha coefficient, developers can ask: which items supply information at which trait levels? Which response categories function? Which items exhibit differential functioning? Can a short form preserve information near the region where decisions are made? Can an item bank support adaptive administration?

The SAPA Project, or Synthetic Aperture Personality Assessment, is a major example of modern large-scale psychometric data collection. Condon and Revelle’s 2015 SAPA data paper reports that approximately 24,000 individuals were administered random subsets of 696 items from 92 public-domain personality scales using the SAPA method between December 2013 and July 2014. ([SAPA Project dataset paper, 2015]) Journal of Open Psychology Data A later SAPA update describes data collected from July 2014 through February 2017, representing approximately 1,150 days of uninterrupted cross-sectional survey data collection using SAPA; it describes SAPA as a collaborative online tool for assessing personality domains including temperament, cognitive abilities, and interests. ([SAPA Project update, 2017]) Journal of Open Psychology Data

The public-domain IPIP ecosystem is adjacent but should not be misdescribed. The official International Personality Item Pool site says IPIP includes over 3,000 items and more than 250 scales, all in the public domain and available to copy, edit, translate, or use without permission or payment. ([Official IPIP site]) ipip.ori.org SAPA used and analyzed large pools of public-domain personality items; it did not originally “create IPIP” as a whole. A more accurate statement is that IPIP supplied a public-domain item-bank infrastructure, while SAPA provided a scalable online sampling and validation method for administering overlapping random subsets of such items and estimating their structure.

This distinction matters because IRT and related latent-variable methods do not remove the need for construct stewardship. A public-domain item pool is not automatically a valid measurement instrument. The item bank must be curated, calibrated, checked for dimensionality, monitored for drift, and validated against the intended use.

8. Computerized adaptive testing: IRT as an engine for item selection

Computerized adaptive testing, or Computerized Adaptive Testing, depends on item information. The adaptive loop is simple in outline:

Start with a provisional estimate of θ\thetaθ.
Select an item with high information near that estimate, subject to content and exposure constraints.
Observe the response.
Update the θ\thetaθ estimate and its standard error.
Repeat until the test reaches a precision, length, content, or classification stopping rule.

Weiss and Kingsbury’s 1984 work on CAT in educational settings described applications that use IRT to select questions from an item pool to estimate a student’s achievement level and the precision of that estimate. ([Weiss & Kingsbury 1984 record]) Experts@Minnesota Lord’s 1980 book also treated information functions, flexilevel tests, multilevel tests, tailored testing, mastery testing, equating, item bias, and ability estimation as practical IRT applications. ([Lord 1980, ERIC record]) ERIC

The conceptual importance is that an item is not valuable in the abstract. It is valuable relative to a current uncertainty state. An easy item is informative for low-θ\thetaθ examinees and nearly useless for high-θ\thetaθ examinees who will almost certainly answer it correctly. A very hard item is informative for high-θ\thetaθ examinees and nearly useless for low-θ\thetaθ examinees who will almost certainly miss it. CAT exploits this by spending items where they reduce uncertainty.

The same principle can be applied to AI evaluation. If a model is clearly below the pass threshold, very hard prompts are wasteful. If a frontier model saturates common benchmarks, easy prompts add little information. A calibrated adaptive benchmark would select tasks near the model’s estimated competence frontier. That is a direct IRT idea.

9. Limitations and failure modes

IRT is not a validity machine. It is a statistical framework whose usefulness depends on modeling assumptions and design quality.

9.1 Unidimensionality

Most basic IRT models assume one dominant latent trait. This is plausible for narrowly defined domains and questionable for broad constructs. “Math ability,” “AI coding ability,” “helpfulness,” “clinical severity,” and “reasoning” are often multidimensional. If the test is multidimensional but modeled as unidimensional, item parameters can become artifacts of the sampled population and item mix.

In AI evaluation, unidimensionality is usually the exception, not the rule. A single “model ability” parameter across coding, factuality, tool use, refusal behavior, long-context synthesis, math, and instruction following is a convenience, not a demonstrated construct.

9.2 Local independence

IRT assumes item responses are independent conditional on θ\thetaθ. This fails when items share passages, templates, solution strategies, leaked training examples, or dependencies in a testlet. In human testing, reading-comprehension passages and item bundles can create local dependence. In LLM evals, templated prompts and benchmark families can create much stronger dependence: solving one item may imply solving dozens of near-duplicates.

9.3 Model fit

The Rasch model may be too restrictive. The 2PL may fit better but sacrifice strict sufficiency and invariance properties. The 3PL may fit multiple-choice data but can estimate unstable guessing parameters. Polytomous models may fail when categories are disordered or raters use scales inconsistently. Better likelihood is not the same as better measurement.

9.4 Sample size and sparse designs

IRT calibration can be data hungry. More flexible models require more observations, especially for stable discrimination and guessing estimates. Sparse matrix designs can work, but only when overlap, anchors, and missingness patterns support identification. The sample-size literature increasingly recommends design-specific simulation rather than fixed folklore thresholds. ([Sage tutorial on IRT sample-size planning]) Sage Journals

9.5 Differential item functioning

If an item behaves differently across groups after conditioning on ability, the item is not invariant. In education, this can involve language, cultural context, disability accommodations, or curriculum exposure. In AI, it can involve model family, training data, prompt format, tokenizer quirks, tool availability, or refusal policies.

9.6 Construct validity

IRT can tell you that an item discriminates among respondents on a latent dimension. It cannot, by itself, prove that the dimension is the construct you intended. A benchmark can have excellent item fit and still measure benchmark familiarity, prompt compliance, test-taking heuristics, or surface style rather than the advertised capability.

9.7 Security, exposure, and contamination

Calibrated item banks are valuable and therefore vulnerable. CAT increases the need for exposure control because high-information items are repeatedly selected. In LLM benchmarks, public item banks can be absorbed into training data, turning difficulty estimates into historical artifacts.

10. Relevance to AI eval design

IRT is directly relevant to AI Evaluation because many current evals reproduce the weaknesses that psychometrics spent a century trying to reduce: unweighted average scores, uncalibrated item difficulty, weak linking across benchmark versions, unmodeled judge variance, and little information about where measurement is precise.

10.1 Mapping IRT concepts to AI evals

Psychometric term	AI-eval analogue	Design implication
Person	Model, agent, system configuration, or policy	Define exactly what is being measured: base model, prompted system, tool-using agent, or deployment pipeline
Ability θ\thetaθ	Latent competence in a domain	Avoid global θ\thetaθ unless construct is narrow or multidimensional model is used
Item	Prompt, task, case, scenario, unit test, preference pair	Estimate difficulty and discrimination rather than treating all tasks equally
Response	Model output or action trace	Use clear scoring rules; preserve partial credit where meaningful
Rater	Human judge, LLM judge, execution harness, unit test	Model judge severity, bias, and reliability when scoring is subjective
Item bank	Calibrated task pool	Track item metadata, provenance, contamination risk, difficulty, discrimination, information
Anchor item	Stable reference prompt/task	Use anchors to link benchmark versions and model generations
Test information	Precision over ability range	Design evals for the decision region: frontier ranking, pass/fail, regression detection, safety threshold

10.2 Anchored rubric calibration

Rubric-based AI evals often fail because rubric categories are not calibrated. A score of “4” from one judge may not mean the same thing as a “4” from another judge or from the same judge under a different prompt. IRT suggests a stricter approach:

Build anchor responses with known or consensus-scored quality.
Include anchor items across evaluation batches.
Estimate item thresholds for rubric categories.
Estimate judge severity and consistency as additional facets.
Detect drift when anchors no longer receive stable scores.

This points toward Many-Facet Rasch Models rather than simple IRT. In many-facet models, item difficulty, respondent ability, rater severity, rubric category thresholds, and sometimes task domains can be estimated jointly. For LLM-as-judge pipelines, this is often the right abstraction: the judge is not a transparent measurement instrument.

10.3 Judge ability and judge bias

LLM-as-judge methods are attractive because human preference data are expensive. Zheng et al.’s MT-Bench and Chatbot Arena paper reports that strong LLM judges can approximate human preferences and achieve over 80% agreement in their setting, but it also explicitly studies limitations including position bias, verbosity bias, self-enhancement bias, and limited reasoning ability. ([Zheng et al., MT-Bench and Chatbot Arena]) arXiv

Psychometrically, a judge is a measurement device with its own error profile. A judge can be severe, lenient, style-biased, position-biased, sensitive to verbosity, or unreliable on specific task types. Treating judge outputs as ground truth without calibration is equivalent to treating an unvalidated test form as a valid scale.

10.4 Item-bank construction for AI

IRT suggests that AI benchmarks should be maintained as item banks rather than static leaderboards. Each item should carry metadata:

Metadata	Why it matters
Construct target	Prevents a generic “reasoning” label from hiding mixed skills
Difficulty estimate	Supports targeting and adaptive selection
Discrimination estimate	Identifies items that separate models well
Guessing or shortcut risk	Flags tasks solvable by artifacts
Information function	Shows where the item improves precision
Template family	Detects local dependence
Contamination risk	Tracks public exposure and training leakage
Scoring method	Separates deterministic correctness from judged quality
Anchor status	Supports linking across benchmark versions
DIF diagnostics	Detects model-family, language, modality, or tool-use artifacts

A good benchmark should contain hard items, but hardness alone is insufficient. An item that every model fails has little ranking information. An item that only weak models solve may be negatively discriminating, which often signals ambiguity, annotation error, shortcut exploitation, or construct mismatch. In AI evals, such items are not rare; they are evidence.

10.5 IRT for NLP and LLM benchmarking

The idea of applying IRT to AI evaluation predates the current LLM wave. Lalor, Wu, and Yu’s 2016 EMNLP paper proposed IRT as an alternative to standard NLP evaluation that treats all test items as equally difficult and equally discriminating; they demonstrated it on Recognizing Textual Entailment and argued that a high accuracy score does not always imply a high IRT score because response patterns and item characteristics matter. ([Lalor et al., EMNLP 2016]) arXiv

Recent LLM-specific work is more ambitious but still early. A 2026 arXiv paper on adaptive testing for LLM evaluation introduces ATLAS, an IRT-based framework using Fisher-information-guided item selection, and reports up to 90% item reduction while preserving measurement precision in its studied benchmarks; it also reports that ability estimates can distinguish among models with similar raw accuracy and shift rankings by more than 10 positions for a substantial minority of evaluated models. ([Adaptive Testing for LLM Evaluation, arXiv]) arXiv

That result is promising, but it should be treated as evidence for a research direction, not proof that IRT invariance holds for LLM evaluation generally.

11. The open question: does IRT transfer cleanly to LLM assessment?

The core unresolved question is whether IRT’s invariance assumptions transfer to LLM-based assessment. The evidence is thin and contested because the “person,” the “item,” and the “rater” are all less stable than in traditional educational testing.

11.1 The “person” is not stable

In human testing, a person’s ability is not perfectly stable, but the construct is at least anchored in an organism at a time. In LLM testing, the “person” might be:

base model weights,
chat-tuned model,
system prompt,
decoding configuration,
tool-using agent,
retrieval-augmented system,
safety policy wrapper,
multi-agent scaffold,
or deployed product pipeline.

Changing any of these may change the response process. A single θ\thetaθ for “GPT-like model ability” is therefore underspecified unless the system boundary is fixed.

11.2 The “item” is not stable

In human testing, item wording matters. In LLM testing, item wording can dominate. Minor prompt changes, formatting, answer-order placement, system messages, examples, refusal instructions, or hidden chain-of-thought constraints can change response probabilities. If item parameters vary across prompt wrappers or model families, the item is not invariant.

11.3 The response process is multidimensional

An LLM item can involve knowledge, retrieval, instruction hierarchy, reasoning, formatting, code syntax, safety behavior, and judge preference simultaneously. A unidimensional θ\thetaθ may fit numerically while concealing that different model families solve the item through different mechanisms.

11.4 The rater is part of the measurement model

In open-ended LLM evals, the response is not self-scoring. It is judged. If the judge is another LLM, then evaluation involves at least three latent components: candidate ability, item difficulty, and judge behavior. LLM-as-judge papers explicitly identify systematic biases such as position and verbosity effects, so the judge cannot be treated as a neutral observation channel. ([Zheng et al., MT-Bench and Chatbot Arena]) arXiv

11.5 Contamination breaks historical calibration

If benchmark items are public, they can enter training corpora. A calibrated difficulty parameter from 2024 may not mean the same thing in 2026 if models have trained on the item, the solution, or near-duplicates. This is not ordinary learning over a school year; it is population-level absorption of the measuring instrument.

11.6 The best near-term stance

The right conclusion is not “IRT does not apply to LLMs.” The right conclusion is: IRT is a useful modeling discipline for LLM evaluation, but its strongest psychometric interpretations require empirical evidence that is usually missing.

For LLM eval design, IRT principles should be used as constraints and diagnostics:

Do not report only average accuracy when item difficulty varies.
Estimate item discrimination and remove or investigate negative-discrimination items.
Use anchor tasks to link benchmark versions.
Model judge effects when using subjective ratings.
Report information curves, not just scalar leaderboard scores.
Treat invariance as an empirical claim to test across model families, prompts, languages, and time.
Prefer multidimensional or many-facet models when the construct or scoring process demands them.

12. Practical design pattern: IRT-informed AI evaluation

A rigorous IRT-inspired AI-eval pipeline would look like this:

Stage	Psychometric action	AI-eval translation
Construct definition	Define latent trait	Specify narrow capability: e.g., Python debugging under unit tests, not “coding”
Item writing	Build item pool	Generate tasks with provenance, metadata, and contamination controls
Pilot calibration	Collect response matrix	Run diverse models and human baselines across overlapping item subsets
Model fitting	Estimate item/person parameters	Fit Rasch/2PL/polytomous/multidimensional models as justified
Fit diagnostics	Check assumptions	Inspect local dependence, DIF, negative discrimination, saturation
Anchor selection	Preserve scale	Choose stable items for future linking
Information targeting	Design forms	Assemble evals for frontier ranking, threshold decisions, or regression testing
Judge calibration	Model rater behavior	Include human/LLM judge anchors, severity estimates, and disagreement analysis
Reporting	Report uncertainty	Publish θ\thetaθ, standard errors, information curves, and item-bank coverage
Maintenance	Monitor drift	Recalibrate after model generations, contamination, or task distribution shifts

This is slower than throwing 500 prompts into a leaderboard and averaging scores. It is also more honest. Average accuracy answers “What fraction of this arbitrary item set did the system pass?” IRT asks “What can we infer about latent competence, with what uncertainty, from these item responses?”

13. Source map

Foundational psychometrics: Lord and Novick’s 1968 Statistical Theories of Mental Test Scores is the canonical reference point for the transition from classical test theory to modern latent-trait measurement; Rasch’s 1960 monograph supplies the foundational probabilistic one-parameter model; Lord’s 1980 book connects IRT to practical testing problems including item analysis, information functions, tailored testing, equating, and item bias. ([Lord & Novick 1968 review]; [Rasch 1960 review record]; [Lord 1980, ERIC record]) Sage Journals+2Cambridge University Press & Assessment+2

Model-family and psychological measurement sources: Embretson and Reise provide a standard treatment for psychologists, including contemporary IRT models, estimation methods, software, and polytomous models; Samejima and Masters are key references for graded-response and partial-credit extensions. ([Embretson & Reise]; [Samejima graded-response monograph record]; [Partial Credit Model, Springer record]) Google Books+2Cambridge University Press & Assessment+2

Operational assessment sources: College Board’s digital SAT technical manual documents MST routing, IRT-based construction/scoring, and use of TIFs/TCCs; ETS’s GRE compendium documents IRT’s role in frequent delivery and the 2PL/3PL model choice; NAEP documentation describes IRT scaling across item blocks and different IRT models for multiple item types. ([College Board Digital SAT Technical Manual]; [ETS GRE research compendium]; [NAEP score-scale documentation]) research.collegeboard.org+2ETS+2

SAPA/IPIP sources: the SAPA data papers document random-subset online administration of public-domain personality items and large cross-sectional samples; the official IPIP site documents the public-domain item pool and scale inventory. ([SAPA Project dataset paper, 2015]; [SAPA Project update, 2017]; [Official IPIP site]) Journal of Open Psychology Data+2Journal of Open Psychology Data+2

AI-evaluation sources: Lalor et al. introduced IRT for NLP test-set evaluation; Zheng et al. studied LLM-as-judge methods and their biases; recent adaptive-testing work applies Fisher-information-guided IRT selection to LLM benchmark reduction and ability estimation, but remains early evidence rather than settled measurement theory. ([Lalor et al., EMNLP 2016]; [Zheng et al., MT-Bench and Chatbot Arena]; [Adaptive Testing for LLM Evaluation, arXiv]) arXiv+2arXiv+2

Companion entries

Core theory: Classical Test Theory, Latent Variable Models, Rasch Model, Two-Parameter Logistic Model, Three-Parameter Logistic Model, Item Characteristic Curve, Test Information Function, Measurement Invariance, Differential Item Functioning, Local Independence

Psychometrics practice: Computerized Adaptive Testing, Multistage Adaptive Testing, Item Bank Calibration, Score Equating, Many-Facet Rasch Models, Polytomous IRT Models, Synthetic Aperture Personality Assessment, International Personality Item Pool

AI evaluation: AI Evaluation, LLM-as-a-Judge, Benchmark Contamination, Adaptive Benchmarking, Rubric Calibration, Judge Reliability, Model Ability Estimation, Evaluation Item Banks

Counterarguments and cautions: Validity Is Not Reliability, Goodhart’s Law in Benchmarks, Construct Underrepresentation, Benchmark Overfitting, Multidimensional Ability, Measurement Drift