A study of what LLMs are actually good and bad at

Not a scoreboard — a set of directional, scoped claims, each tied to the sources that support or contest it.

A ratchet is a fix that cannot slip back: placed in structure, it applies on every later run, while a fix given in conversation dies with the session. That is what this catalog is for and what it is named after. The ratchet collects the 17 sources on systems that try to build one, and the 20 techniques that decide what a loop keeps.

Capabilities are the topics. Claims are the findings: one scoped statement, with the sources that support or contest it. Techniques are the fixes; whether a fix works is itself a claim. Sources are the evidence. Heuristics are borrowed or distilled rules, tested the same way.

Every claim says who has read it: 572 of 657 were reviewed by AI, the rest by AI and a person. A person’s review is what lets a claim count toward a technique’s standing. How this is collected.

Try:
50
capabilities
657
claims
686
sources
340
techniques
25
heuristics

    Searches everything in the catalog. Runs in your browser.

    Worth a look

    2026-10-19 · the hypothesis this site exists to test: models trained on a literature that hides its failures inherit its optimism. Argued, not yet measured — see the Ratchet page for the experiment
    LLMs Have Made Failure Worth Publishing
    Sungmin Lee, 2026

    The case here is argued, not measured: publication bias toward positive results is claimed to propagate into LLMs, which inherit a distorted prior on how often experiments succeed, just as the corpus of unpublished failure becomes the largest untapped source of human-generated training text.

    Claims drawn from it

    the source in full · featured as something worth reading, not as something judged correct

    Most recently checked claims

    All claims →

    Reasoning

    Multi-step reasoning, planning, decomposition, mathematical reasoning.

    • Carrying a chain of inference through to its conclusion

      Whether a model can hold several dependent inference steps and reach the conclusion they entail, as opposed to pattern-matching the answer; failures show as correct steps with a wrong join, or shortcuts.

    • Deducing validly from logical structure

      Covers strict rule-based deduction, stability under label-preserving rephrasings, and retracting conclusions when evidence changes; excludes arithmetic and factual QA.

    • Digit-level arithmetic

      Multi-digit arithmetic, counting, and symbol manipulation fail in ways tied to tokenization and memorization.

    • Formalizing and machine-checking mathematics

      Covers autoformalization into a solver-checkable language, producing proof-assistant-verified proofs, and localizing the first flawed step in a proof; excludes informal math word problems.

    • Planning a sequence of actions before acting

      Whether a model can lay out an ordered set of steps toward a goal, with the dependencies right, before or instead of executing them; distinct from carrying the plan out.

    • Predicting future events

      Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts.

    Knowledge

    Factual recall, domain knowledge, freshness, and stating false facts.

    • Absorbing and flexibly using new knowledge

      Covers whether newly injected or post-training facts can be recalled under paraphrase, combined, and used to override stale beliefs; excludes retrieval-based grounding at inference time.

    • Checking claims against evidence

      Models accept confident false premises and resolve conflicts between sources by majority or recency rather than by reliability.

    • Competence in a specialist domain

      Whether a general model performs like a practitioner in a specific field (medicine, law, hardware design, a scientific subfield), and where the general-purpose competence stops.

    • Not generalizing "A is B" to "B is A"

      A model that learned a fact in one direction often cannot recall it when asked in the reverse direction.

    • Recalling facts it was trained on

      Whether a model retrieves a fact from its weights when asked, especially long-tail facts; the failure is a wrong or missing answer, measured against ground truth, not the confidence of a made-up one.

    • Stating false facts confidently

      Models generate plausible but false statements, citations, and details, with no signal that they are unsure.

    Coding

    Code generation, debugging, repository navigation, software architecture, secure code.

    • Finding and fixing a fault in existing code

      Whether a model can localise the cause of a failing behaviour in code it did not just write and repair it without breaking something else; distinct from generating code and from self-repair of its own output.

    • Finding what matters in a large codebase

      Whether an agent can locate the files, symbols and call paths relevant to a task in a repository too large to read, and avoid editing the wrong place; the failure is a plausible patch in the wrong file.

    • Generating and editing working code

      Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.

    • Making design decisions that hold up as a system grows

      Whether a model can choose structure — module boundaries, interfaces, data flow — that stays maintainable, as opposed to code that passes the tests today; the failure is locally correct code that raises the cost of every later change.

    • Writing a reusable locator: regex, SQL, XPath

      Whether a model can turn an intent or a few examples into a grep pattern, regex, SQL query, XPath or jq path that selects the right subset, so the pattern runs over the whole collection without calling the model again.

    • Writing secure code and dependencies

      Generated code carries known vulnerability patterns, and models suggest packages that do not exist.

    Agentic

    Tool use, instruction following, long-horizon execution, self-correction, environment interaction.

    Context

    Long-context comprehension, retrieval utilization, noise tolerance, memory and state.

    Verification

    Error detection, critique, and uncertainty calibration — the model checking work, its own or others'.

    • Biased when judging other outputs

      A model used as a judge favors whichever answer is shown first, longer answers, and its own outputs.

    • Judging another's work as well as a careful reviewer would

      The model as critic or judge: whether its evaluation of an artifact agrees with careful human judgement and is actionable; the failures are bias toward surface features, agreement with the author, and vague critique that sounds specific.

    • Saying it does not know

      Covers detecting unanswerable or underspecified problems and gaps between what hidden states encode and what the model asserts; excludes ordinary accuracy on solvable problems.

    • Spotting an error in a given artifact

      Whether a model can find the mistake in something put in front of it — a proof, a program, an argument, a table — when told there may be one; distinct from fixing it, and from judging overall quality.

    Behavior

    Sycophancy, deception, faithfulness of explanations, diversity, and disparities across languages and cultures.

    • Changing what a person believes or decides

      How effectively a model shifts a person's beliefs or choices in conversation, relative to a human doing the same; the risk is the flip side of the capability, a system that can talk its operators out of constraining it.

    • Cutting to the chase proposed

      Giving the answer directly when that's what is asked, without preamble, restatement or visible working the reader didn't request; the failure is padding the model was told to skip, or skipping working the task needed.

    • Gaming the measure instead of doing the task proposed

      Whether a model trained against a reward or a monitor learns the intended behaviour or learns to score well some other way, including ways the monitor cannot see.

    • Handling culture-specific norms and viewpoints

      Covers implicit cultural norms, dialects, local pragmatics, and representing the spread of a population's views; excludes general multilingual fluency or factual trivia about cultures.

    • Performing evenly across languages

      Covers gaps between high- and low-resource languages on identical tasks, including uneven parametric knowledge coverage by region; excludes translation quality per se.

    • Retaining skills and safety after fine-tuning

      Covers loss of general capability, out-of-domain skill, refusal behavior, or output diversity caused by domain, multimodal, alignment or RL post-training; excludes in-domain accuracy gains and inference-time behavior shifts.

    • Saying what actually drove the answer

      Whether a model's stated reasoning reflects the computation that produced its answer, rather than a plausible story told after the fact.

    • Strategic deception and detecting it

      Whether a model can deliberately induce false beliefs in others when a task calls for it, and detect deception from others.

    • Telling the user what they want to hear

      Models shift answers toward the user's stated view, cave to pushback, and mirror mistaken premises.

    Security

    Prompt injection and jailbreak robustness.

    • Following instructions hidden in data

      Models treat instructions found in retrieved documents, tool outputs, or web pages as if they came from the user.

    • Refusing harmful requests under obfuscation

      Covers maintaining refusal when harmful intent is wrapped in roleplay or encodings, harm that emerges only from combined segments, and unsafe content inside reasoning traces; excludes instructions injected via retrieved data.

    Evaluation

    Whether benchmarks and evaluation protocols measure what they claim.

    Perception

    Audio, video, images and the physical world.