a-single-aggregate-benchmark-conceals-domain-trajectories-that-differ-in-shape CLM-0009mechanismsingle paperReviewed by AI
When each capability domain's annual benchmark frontier is normalised to its own scale (0 at the benchmark's entry-year frontier, 100 at a perfect score), the domains move on different curves — broad knowledge gains slow from 32.8 to 26.9 to 17.6 points a year, legal reasoning from 48.2 to 11.4 to 4.9, while advanced mathematics accelerates from 32.8 to 53.6 and multimodal reasoning jumps 59.7 then only 2.5 — so a single aggregate benchmark score conceals both the level and the trajectory shape of progress, and cannot say which capabilities are saturating.
Capability: Whether the measurement made the finding
Sources
- Section 2.1, Observation 1, with the square-root family weighting the authors use so better-covered benchmark families count more without dominating a domain. The claim is about what aggregation hides; the specific increments are this paper's construction.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across 393 model–benchmark observations in ten capability domains for models released between 2023 and September 2026, the share of benchmark headroom closed is uneven: bounded, readily verified domains are near their ceilings (advanced mathematics 86.4 and graduate-level science 85.8 on a 0–100 index where 0 is the benchmark's entry-year frontier), while interactive, stateful ones lag (software engineering 52.6, search and terminal agents 56.8, tool agents 39.9) — so gains in bounded environments have not transferred uniformly to long, stateful workflows where errors propagate across the trajectory.Completing tasks that take many steps or many hours · reviewed by AI
- On a suite of verifier-checked tool-using ReAct task families (arithmetic, SQL analytics, security judgment, URL grounding, dependency planning, orchestration, tool policy), a single task-weighted aggregate accuracy hides inverted per-family rankings: the model that ranks last overall can lead both URL-grounding families while scoring lowest on structured SQL and multi-step tool families, which spread the four tested models by more than sixty points.Whether the measurement made the finding · reviewed by AI
- On 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.Competence in a specialist domain · reviewed by AI
- On repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.Completing tasks that take many steps or many hours · reviewed by AI
- For embodied vision-language models trained as separate capability specialists (spatial, temporal, action guidance, state verification) from one backbone, consolidating them by TIES weight merging and then routed multi-teacher on-policy distillation retains more of each specialist's gain than either step alone or than mixed-data RL, which shows capability-specific biases.Turning plans into physical action · reviewed by AI
Notes
Relevant to this catalog's own design: capabilities are not scored, and this is one reason. It also bears on the backtest, which measures the catalog on one number.