Quantifying construct validity in large language model evaluations

Abstract

Verbatim from the arXiv abstract page (fetched 2026-09-11):

The LLM community often reports benchmark results as if they are synonymous with general model capabilities. However, benchmarks can have problems that distort performance, like test set contamination and annotator error. How can we know that a benchmark is a reliable indicator of some capability that we want to measure? This question concerns the construct validity of LLM benchmarks, and it requires separating benchmark results from capabilities when we model and predict LLM performance. Both social scientists and computer scientists propose formal models - latent factor models and scaling laws - for identifying the capabilities underlying benchmark scores. However, neither technique is satisfactory for construct validity. Latent factor models ignore scaling laws, and as a result, the capabilities they extract often proxy model size. Scaling laws ignore measurement error, and as a result, the capabilities they extract are both uninterpretable and overfit to the observed benchmarks. This thesis presents the structured capabilities model, the first model to extract interpretable and generalisable capabilities from a large collection of LLM benchmark results. I fit this model and its two alternatives on a large sample of results from the OpenLLM Leaderboard. Structured capabilities outperform latent factor models on parsimonious fit indices, and exhibit better out-of-distribution benchmark prediction than scaling laws.

Key Contributions

• Names the trade the two existing formalisms make. Latent factor models (from psychometrics) model measurement error but ignore scale, so the "capability" they recover is largely a proxy for parameter count. Scaling laws model scale but assume the benchmark measures without error, so the capability they recover is uninterpretable and overfits the observed benchmark set.
• The structured capabilities model combines both: latent factors with a scaling-law structure imposed on the factor, so capability is identified separately from both measurement error and model size.
• Fit on real leaderboard data at scale rather than on a synthetic demonstration.
• Two evaluations: parsimonious fit (does it explain the observed scores without buying fit with parameters) and out-of-distribution prediction (does it predict a benchmark it never saw).

Methodology

Sample: OpenLLM Leaderboard v2, 4,576 models, reduced to 4,395 after dropping 181 with incomplete results; 19 BBH subtasks retained from 23. Four model specifications are fitted and compared — pure EFA, structured, pure logistic, structured logistic — with standard SEM fit indices, plus a held-out-subtask prediction experiment against an observational (PCA-based) scaling law.

Results

Experiment A — model fit (values as read from the paper's Table 2 via the arXiv HTML, 2026-09-11):

The logistic transformation is where the fit gain lives: RMSEA falls from 0.0876 (mediocre) to 0.0623 (conventionally acceptable-to-good). The structured-logistic column was not legible in the fetched table.

Experiment B — out-of-distribution prediction. The structured capabilities model attains lower mean squared error on held-out subtasks than observational scaling laws. The per-subtask MSE values were not legible in the fetched HTML table — see caution below.

Limitations

• This is a thesis, not a peer-reviewed paper, and it is dated 2026-02-17, well outside the freshness window of this pass. It is taken because evals had never ingested and construct validity is the rung's founding question.
• The headline OOD claim is not yet quantified in this file. The abstract states "better out-of-distribution benchmark prediction than scaling laws" and the HTML table for Experiment B did not render its numbers on fetch. Treat the OOD result as directional and unquantified here; do not cite a magnitude for it. Anything compiled from this source must cite the Experiment A fit indices, which are read and recorded above, not the OOD comparison.
• The empirical base is narrow in benchmark terms — 19 BBH subtasks from one leaderboard. Breadth is in models (4,395), not in benchmark families, so the construct being validated is BBH-shaped.
• AIC/BIC are negative for two specifications and positive for the third, which means the likelihood is on different scales between the untransformed and logistic forms; the information criteria are not comparable across that boundary and should not be read as a ranking.

Why this is on the evals rung

E2 (the tool-calling validity audit) is the incident: an instrument whose noise exceeds its findings. This is the method for the same question asked generally — given a table of benchmark scores, how would you tell whether the benchmark measures the capability it names, rather than model size? The pairing is deliberate; neither is a bet.

Source: Quantifying construct validity in large language model evaluations by Ryan Othniel Kearns — arXiv:2602.15532, submitted 2026-02-17.