Capabilities
A capability is a topic: one thing a model has to be able to do, such as arithmetic, following a procedure, or fixing its own mistakes. It is a heading, not a finding, and it carries no score. What is known about it lives in the claims filed under it, each with its sources; the techniques listed on it are fixes that address it, and whether a fix works is itself a claim. A capability with no claims yet is a gap to fill, not an error. Some are marked proposed: added from the map or by the ingestion pipeline without anyone deciding it belongs. Those are here to be argued with.
Capabilities are the topics. Claims are the findings: one scoped statement, with the sources that support or contest it. Techniques are the fixes; whether a fix works is itself a claim. Sources are the evidence. Heuristics are borrowed or distilled rules, tested the same way.
| Capability | Claims | Contested | Techniques | Sources |
|---|---|---|---|---|
| Reasoning6 Multi-step reasoning, planning, decomposition, mathematical reasoning. | ||||
| Carrying a chain of inference through to its conclusion Whether a model can hold several dependent inference steps and reach the conclusion they entail, as opposed to pattern-matching the answer; failures show as correct steps with a wrong join, or shortcuts. | 11 | — | 8 | 14 |
| Deducing validly from logical structure Covers strict rule-based deduction, stability under label-preserving rephrasings, and retracting conclusions when evidence changes; excludes arithmetic and factual QA. | 3 | — | 1 | 3 |
| Digit-level arithmetic Multi-digit arithmetic, counting, and symbol manipulation fail in ways tied to tokenization and memorization. also: numerical reasoning, token-level math | 11 | 1 | 3 | 9 |
| Formalizing and machine-checking mathematics Covers autoformalization into a solver-checkable language, producing proof-assistant-verified proofs, and localizing the first flawed step in a proof; excludes informal math word problems. | 5 | — | 2 | 5 |
| Planning a sequence of actions before acting Whether a model can lay out an ordered set of steps toward a goal, with the dependencies right, before or instead of executing them; distinct from carrying the plan out. | 13 | — | 13 | 16 |
| Predicting future events Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts. | 16 | — | 9 | 16 |
| Knowledge6 Factual recall, domain knowledge, freshness, and stating false facts. | ||||
| Absorbing and flexibly using new knowledge Covers whether newly injected or post-training facts can be recalled under paraphrase, combined, and used to override stale beliefs; excludes retrieval-based grounding at inference time. | 6 | — | 4 | 7 |
| Checking claims against evidence Models accept confident false premises and resolve conflicts between sources by majority or recency rather than by reliability. also: epistemic verification, truth adjudication, knowledge conflict | 10 | — | 4 | 9 |
| Competence in a specialist domain Whether a general model performs like a practitioner in a specific field (medicine, law, hardware design, a scientific subfield), and where the general-purpose competence stops. | 10 | — | 5 | 10 |
| Not generalizing "A is B" to "B is A" A model that learned a fact in one direction often cannot recall it when asked in the reverse direction. also: directional recall, asymmetric knowledge | 3 | — | 1 | 2 |
| Recalling facts it was trained on Whether a model retrieves a fact from its weights when asked, especially long-tail facts; the failure is a wrong or missing answer, measured against ground truth, not the confidence of a made-up one. | 4 | — | 1 | 5 |
| Stating false facts confidently Models generate plausible but false statements, citations, and details, with no signal that they are unsure. also: confabulation, fabrication, factual error, making things up, invented citations | 57 | — | 21 | 83 |
| Coding6 Code generation, debugging, repository navigation, software architecture, secure code. | ||||
| Finding and fixing a fault in existing code Whether a model can localise the cause of a failing behaviour in code it did not just write and repair it without breaking something else; distinct from generating code and from self-repair of its own output. | 4 | — | 1 | 6 |
| Finding what matters in a large codebase Whether an agent can locate the files, symbols and call paths relevant to a task in a repository too large to read, and avoid editing the wrong place; the failure is a plausible patch in the wrong file. also: issue localization, fault localization, code localization, bug localization | 6 | — | 6 | 6 |
| Generating and editing working code Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging. | 21 | — | 9 | 26 |
| Making design decisions that hold up as a system grows Whether a model can choose structure — module boundaries, interfaces, data flow — that stays maintainable, as opposed to code that passes the tests today; the failure is locally correct code that raises the cost of every later change. | 3 | — | — | 3 |
| Writing a reusable locator: regex, SQL, XPath Whether a model can turn an intent or a few examples into a grep pattern, regex, SQL query, XPath or jq path that selects the right subset, so the pattern runs over the whole collection without calling the model again. also: locator generation, pattern synthesis, text-to-SQL, regex generation, programming by example, grep, grep-style locators | 17 | — | 15 | 17 |
| Writing secure code and dependencies Generated code carries known vulnerability patterns, and models suggest packages that do not exist. also: insecure code generation, package hallucination, slopsquatting | 8 | — | 3 | 7 |
| Agentic8 Tool use, instruction following, long-horizon execution, self-correction, environment interaction. | ||||
| Completing tasks that take many steps or many hours Whether an agent can carry a task through a long sequence of actions without losing the thread, compounding small errors, or stopping early; measured by how long a task, in human time or in steps, a model can finish reliably. | 23 | — | 17 | 26 |
| Fixing its own mistakes Whether and how a model can diagnose and correct a wrong answer once it's already made. | 15 | 2 | 6 | 17 |
| Following an unfamiliar procedure Models drift from a new multi-step procedure or set of constraints back to familiar routines. also: instruction following, constraint adherence, workflow compliance | 16 | 1 | 3 | 18 |
| Getting more from several models than from one Whether a system of several models or agents — debating, voting, dividing work, or sharing state — does better than its best single member; the failure is a quorum of copies that agree on the same mistake. also: multi-agent systems, ensembles, model quorum, mixture of agents, LLM debate | 38 | — | 32 | 42 |
| Improving its own scaffolding from failures Whether an agent system can read its own failure traces and rewrite the prompts, tools and policies it runs inside so later runs do better, with no person making the edit. also: self-improving agents, harness optimization, scaffold evolution, prompt optimization | 29 | 1 | 24 | 35 |
| Prioritizing safety under conflicting goals When a task goal competes with a safety rule or a required check, models sometimes drop the check to finish the task. also: critical task prioritization, oversight subversion, guardrail bypass | 8 | — | 3 | 7 |
| Turning plans into physical action Closing the loop from perception to action in a body or simulator, where mistakes have physical consequences and cannot be retried freely. | 37 | — | 24 | 37 |
| Using the tools it is given Models often fail to reach for provided tools, skills, or documented procedures, call them with wrong arguments, or act on a stale or misread result. also: tool calling, function calling, API use, skill utilization, computer use, environment interaction, acting on an environment and reading what came back | 36 | 1 | 18 | 38 |
| Context5 Long-context comprehension, retrieval utilization, noise tolerance, memory and state. | ||||
| Keeping its own context clean Agents fill their working context with re-read files, restated notes, and excess documentation until quality degrades. also: context management, over-documentation, context bloat | 4 | — | 2 | 4 |
| Losing information in long inputs Accuracy drops as inputs grow and is worst for information placed in the middle of the context. also: lost in the middle, needle in a haystack, effective context length | 14 | 1 | 6 | 14 |
| Remembering across sessions Assistants fail to recall and apply facts, preferences, and decisions from earlier sessions. also: persistent memory, cross-session memory, personalization | 11 | — | 5 | 12 |
| Tracking state through a long task Models lose track of the current state of entities, files, or an environment after many steps. also: entity tracking, world-model consistency, stale state | 5 | — | 3 | 5 |
| Using what was retrieved, and only that Given relevant passages in context, whether a model actually uses them, prefers them over its own prior when they conflict, and ignores the irrelevant ones; the failure is an answer the passages do not support. | 1 | — | — | 1 |
| Verification4 Error detection, critique, and uncertainty calibration — the model checking work, its own or others'. | ||||
| Biased when judging other outputs A model used as a judge favors whichever answer is shown first, longer answers, and its own outputs. also: LLM-as-judge bias, position bias, verbosity bias, self-enhancement bias | 3 | — | 2 | 2 |
| Judging another's work as well as a careful reviewer would The model as critic or judge: whether its evaluation of an artifact agrees with careful human judgement and is actionable; the failures are bias toward surface features, agreement with the author, and vague critique that sounds specific. also: LLM-as-a-judge, model as critic, critique generation, reward modelling | 12 | — | 8 | 13 |
| Saying it does not know Covers detecting unanswerable or underspecified problems and gaps between what hidden states encode and what the model asserts; excludes ordinary accuracy on solvable problems. also: abstention, knowing what it does not know, unanswerable questions, selective prediction, calibration | 25 | — | 19 | 23 |
| Spotting an error in a given artifact Whether a model can find the mistake in something put in front of it — a proof, a program, an argument, a table — when told there may be one; distinct from fixing it, and from judging overall quality. | 8 | — | 5 | 8 |
| Behavior9 Sycophancy, deception, faithfulness of explanations, diversity, and disparities across languages and cultures. | ||||
| Changing what a person believes or decides How effectively a model shifts a person's beliefs or choices in conversation, relative to a human doing the same; the risk is the flip side of the capability, a system that can talk its operators out of constraining it. also: persuasiveness, influence, manipulation | 7 | — | 2 | 8 |
| Cutting to the chaseproposed Giving the answer directly when that's what is asked, without preamble, restatement or visible working the reader didn't request; the failure is padding the model was told to skip, or skipping working the task needed. also: answer-only compliance, no-thinking, concision, verbosity, overthinking | 4 | — | 3 | 3 |
| Gaming the measure instead of doing the taskproposed Whether a model trained against a reward or a monitor learns the intended behaviour or learns to score well some other way, including ways the monitor cannot see. also: reward hacking, specification gaming, reward tampering, Goodhart | 3 | — | 2 | 2 |
| Handling culture-specific norms and viewpoints Covers implicit cultural norms, dialects, local pragmatics, and representing the spread of a population's views; excludes general multilingual fluency or factual trivia about cultures. | 5 | — | 2 | 6 |
| Performing evenly across languages Covers gaps between high- and low-resource languages on identical tasks, including uneven parametric knowledge coverage by region; excludes translation quality per se. | 1 | — | — | 1 |
| Retaining skills and safety after fine-tuning Covers loss of general capability, out-of-domain skill, refusal behavior, or output diversity caused by domain, multimodal, alignment or RL post-training; excludes in-domain accuracy gains and inference-time behavior shifts. also: catastrophic forgetting, alignment tax, mode collapse, output diversity, exploration collapse | 8 | — | 7 | 11 |
| Saying what actually drove the answer Whether a model's stated reasoning reflects the computation that produced its answer, rather than a plausible story told after the fact. | 8 | — | 4 | 7 |
| Strategic deception and detecting it Whether a model can deliberately induce false beliefs in others when a task calls for it, and detect deception from others. | 7 | — | 3 | 6 |
| Telling the user what they want to hear Models shift answers toward the user's stated view, cave to pushback, and mirror mistaken premises. also: flattery, agreement bias, caving to pushback | 15 | — | 3 | 11 |
| Security2 Prompt injection and jailbreak robustness. | ||||
| Following instructions hidden in data Models treat instructions found in retrieved documents, tool outputs, or web pages as if they came from the user. also: indirect prompt injection, jailbreak via content | 19 | — | 6 | 20 |
| Refusing harmful requests under obfuscation Covers maintaining refusal when harmful intent is wrapped in roleplay or encodings, harm that emerges only from combined segments, and unsafe content inside reasoning traces; excludes instructions injected via retrieved data. | 3 | — | 1 | 4 |
| Evaluation1 Whether benchmarks and evaluation protocols measure what they claim. | ||||
| Whether the measurement made the finding Whether a reported result describes the model or the way it was scored — the metric, the prompt format, the contamination, the baseline. | 26 | — | 6 | 32 |
| Perception3 Audio, video, images and the physical world. | ||||
| Grounding answers in the image Saying only what the image supports, rather than describing plausible objects that are not there. | 29 | — | 16 | 26 |
| Reasoning about time in video Tracking motion, ordering, causality, and change across video frames rather than treating video as isolated stills. | 19 | — | 10 | 20 |
| Understanding speech and audio Hearing what is said and how it is said — tone, emotion, speaker identity, and events in a soundscape. | 10 | — | 4 | 11 |