Capabilities

Topics, not scores — each lists the claims filed under it. Some are marked proposed: the ingestion pipeline found several papers converging on the same competence and added it, without anyone deciding it belongs. Those are here to be argued with.

Capability	Tags	Claims	Contested	Status
Absorbing and flexibly using new knowledge Covers whether newly injected or post-training facts can be recalled under paraphrase, combined, and used to override stale beliefs; excludes retrieval-based grounding at inference time.	general	—	—	proposed
Biased when judging other outputs A model used as a judge favors whichever answer is shown first, longer answers, and its own outputs.	evaluation, llm-as-judge	3	—	active
Checking claims against evidence Models accept confident false premises and resolve conflicts between sources by majority or recency rather than by reliability.	chat-assistant, data-analysis, knowledge, rag-qa	7	—	active
Deducing validly from logical structure Covers strict rule-based deduction, stability under label-preserving rephrasings, and retracting conclusions when evidence changes; excludes arithmetic and factual QA.	general	—	—	proposed
Digit-level arithmetic Multi-digit arithmetic, counting, and symbol manipulation fail in ways tied to tokenization and memorization.	data-analysis, general, math, reasoning	8	1	active
Fixing its own mistakes Whether and how a model can diagnose and correct a wrong answer once it's already made.	agentic, coding-agent, math	10	2	active
Following an unfamiliar procedure Models drift from a new multi-step procedure or set of constraints back to familiar routines.	agentic, autonomous-agent, chat-assistant, coding-agent, customer-support	10	1	active
Following instructions hidden in data Models treat instructions found in retrieved documents, tool outputs, or web pages as if they came from the user.	autonomous-agent, coding-agent, customer-support, rag-qa, security	11	—	active
Formalizing and machine-checking mathematics Covers autoformalization into a solver-checkable language, producing proof-assistant-verified proofs, and localizing the first flawed step in a proof; excludes informal math word problems.	general	—	—	proposed
Generating and editing working code Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.	reasoning, agentic	7	—	active
Grounding answers in the image Saying only what the image supports, rather than describing plausible objects that are not there.	knowledge, behavior	9	—	active
Handling culture-specific norms and viewpoints Covers implicit cultural norms, dialects, local pragmatics, and representing the spread of a population's views; excludes general multilingual fluency or factual trivia about cultures.	general	—	—	proposed
Keeping its own context clean Agents fill their working context with re-read files, restated notes, and excess documentation until quality degrades.	autonomous-agent, coding-agent, context	4	—	active
Keeping outputs and solution paths diverse Covers mode collapse, entropy decline, and repeated attempts retreading the same strategy after sampling, fine-tuning, or RL; excludes single- answer accuracy and stylistic variety alone.	general	—	—	proposed
Losing information in long inputs Accuracy drops as inputs grow and is worst for information placed in the middle of the context.	coding-agent, context, long-document, rag-qa	7	1	active
Not generalizing "A is B" to "B is A" A model that learned a fact in one direction often cannot recall it when asked in the reverse direction.	general, knowledge	3	—	active
Performing evenly across languages Covers gaps between high- and low-resource languages on identical tasks, including uneven parametric knowledge coverage by region; excludes translation quality per se.	general	—	—	proposed
Predicting future events Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts.	reasoning, knowledge	2	—	active
Prioritizing safety under conflicting goals When a task goal competes with a safety rule or a required check, models sometimes drop the check to finish the task.	agentic, autonomous-agent, coding-agent, customer-support	7	—	active
Reasoning about time in video Tracking motion, ordering, causality, and change across video frames rather than treating video as isolated stills.	reasoning, context	1	—	active
Refusing harmful requests under obfuscation Covers maintaining refusal when harmful intent is wrapped in roleplay or encodings, harm that emerges only from combined segments, and unsafe content inside reasoning traces; excludes instructions injected via retrieved data.	general	—	—	proposed
Remembering across sessions Assistants fail to recall and apply facts, preferences, and decisions from earlier sessions.	autonomous-agent, chat-assistant, context, customer-support	8	—	active
Retaining skills and safety after fine-tuning Covers loss of general capability, out-of-domain skill, or refusal behavior caused by domain, multimodal, or alignment fine-tuning; excludes in-domain accuracy gains and inference-time behavior shifts.	general	—	—	proposed
Saying it does not know Covers detecting unanswerable or underspecified problems and gaps between what hidden states encode and what the model asserts; excludes ordinary accuracy on solvable problems.	general	—	—	proposed
Saying what actually drove the answer Whether a model's stated reasoning reflects the computation that produced its answer, rather than a plausible story told after the fact.	behavior, evaluation	—	—	active
Stating false facts confidently Models generate plausible but false statements, citations, and details, with no signal that they are unsure.	chat-assistant, general, knowledge, rag-qa	22	—	active
Strategic deception and detecting it Whether a model can deliberately induce false beliefs in others when a task calls for it, and detect deception from others.	behavior	3	—	active
Telling the user what they want to hear Models shift answers toward the user's stated view, cave to pushback, and mirror mistaken premises.	behavior, chat-assistant, customer-support, llm-as-judge	11	—	active
Tracking state through a long task Models lose track of the current state of entities, files, or an environment after many steps.	autonomous-agent, coding-agent, context, data-analysis	3	—	active
Turning plans into physical action Closing the loop from perception to action in a body or simulator, where mistakes have physical consequences and cannot be retried freely.	agentic, reasoning	—	—	active
Understanding speech and audio Hearing what is said and how it is said — tone, emotion, speaker identity, and events in a soundscape.	knowledge, reasoning	—	—	active
Using the tools it is given Models often fail to reach for provided tools, skills, or documented procedures, or call them with wrong arguments.	agentic, autonomous-agent, coding-agent, customer-support	14	—	active
Whether the measurement made the finding Whether a reported result describes the model or the way it was scored — the metric, the prompt format, the contamination, the baseline.	evaluation	10	—	active
Writing secure code and dependencies Generated code carries known vulnerability patterns, and models suggest packages that do not exist.	coding-agent, security	7	—	active