About
This is an index of what language models are good and bad at, organised by capability rather than by what is new. Every claim links to the sources it came from, and where the evidence disagrees, both sides are kept.
Capabilities are the topics. Claims are the findings: one scoped statement, with the sources that support or contest it. Techniques are the fixes; whether a fix works is itself a claim. Sources are the evidence. Heuristics are borrowed or distilled rules, tested the same way.
Every entry records who has read it — a model, or a model and a person — so you can judge any entry for yourself, because how it got here is written on it.
Why not a paper index
Papers with Code and paper feeds tell you which papers exist about a topic. This tells you what they found, and under what conditions it stops being true. The unit here is a claim, not a paper: a scoped statement with the papers for and against attached and a falsifier stated. Anyone, human or agent, can add support or contest it. Techniques take their standing from those claims, so “does this fix work?” returns evidence rather than a reading list. When a claim breaks, that is kept as the most valuable entry on the site, not removed.
Where entries come from
A nightly job queries OpenAlex for new arXiv papers, filters them to work that is plausibly about a capability tracked here, and asks a model to judge whether each paper is genuinely about that capability and whether it supports or cuts against something already filed. Papers that survive get read in full, and a model drafts a proposed claim from them.
Those claims enter the catalog marked Reviewed by AI. They appear in lists, counts, search and the contested view like anything else, because an index is useful as soon as it exists. The one thing a person’s review adds is weight: only claims marked Reviewed by AI and a person decide whether a technique is judged to work, or count in the internal scorecard that tracks whether this catalog is getting things right.
Other entries were filed by hand, usually from well-known papers, when the structure of the catalog was being worked out.
Right now: 572 of 657 claims are reviewed by AI, and 85 by AI and a person. Many will stay AI-reviewed. There is more worth indexing than one person can read, so that is a normal permanent state here, not a queue waiting to be cleared.
What a claim carries
- Its scope, inside the sentence. Which models, which tasks, which setup. A claim without its scope condition gets misapplied, so the condition is written into the statement rather than appended.
- Whether it is durable or perishable. A mechanism claim explains why something happens and tends to outlive a model generation. An observation describes how some model or era behaves and is expected to go stale.
- How well it is backed — one paper, replicated across independent work, argued from how something works without anyone measuring it, or someone’s own observation. This is a category, not a score, and nothing here is ranked.
- Its sources, and their side. Each source is marked as supporting or contesting. A contested claim keeps both, plus a written guess at why the evidence disagrees.
What sources are, and what they are worth
| Kind | Count | Treated as |
|---|---|---|
| paper | 668 | Published or preprint work. Titles are checked against arXiv. |
| vendor-doc | 3 | Guidance from whoever builds the model. Privileged about its own product, weak on efficacy (the evaluations are not visible), and interested. Never counted as replication. |
| post | 14 | Informal writing. Good for what practitioners believe and for negative results nobody publishes. Can raise a claim; cannot strongly back one. |
| observation | 1 | Something noticed directly rather than read. |
Anything that can be edited or deleted after the fact — a post, a vendor page, a PDF served from a repository — is archived here verbatim, and a nightly check re-reads the original and reports when it stops matching. A citation should not be able to quietly stop meaning what it meant.
What this gets wrong
- Coverage is uneven. Some capabilities have several claims and some have none. A thin page means nobody has filed here yet, never that there is nothing to know.
- Machine-drafted claims can be wrong in confident-sounding ways. Numbers in a draft are checked against the source text automatically, and anything that fails is flagged on the claim — but a fluent, plausible, mistaken summary passes that check.
- Almost nothing here has been contested by anyone outside. Both sides of a contested claim were assembled by the same reader. That is the weakest part of the whole thing.
- The index is one person’s reading. What gets filed reflects what one person went looking for, which is a bias no amount of structure removes.
None of that is a reason to present the contents with more confidence than they have. It is the reason every entry shows its sources, its backing, and how it arrived — so you can decide, rather than being asked to trust.
For agents: the MCP server
Without MCP, fetch /index.json for the whole catalog in one file, or /llms.txt for a plain-text description of the records.
The catalog is meant to be asked, not browsed. It is served to Claude Code, or any MCP client, by a local server with seven tools. The one to start with is advise: describe a situation — the failure you see, what your environment has — and it returns the techniques that fit, each with its standing, the conditions it needs, what it costs, when it fails, and the evidence against it. Ranked categorically, never scored.
The other six: search (words and meaning together), get, technique_standing, related_claims, list, and open_questions, which returns the research briefs. Meaning search runs a small local model; there is no API key and no cost.
It runs from a checkout of the catalog. Clone github.com/Russ-Miller/RSIratchet, run npm install, then register it once:
claude mcp add -s user rsiratchet -- npx --prefix /path/to/RSIratchet tsx /path/to/RSIratchet/scripts/mcp-server.mtsThen ask, in any session: “does rsiratchet have a technique for a quorum of several models?” The honest answer today is no, with two heuristics that explain why, which is the kind of answer this is built to give. A hosted version at a fixed URL is on the roadmap.
Everything is plain YAML in a public git repository, so the history of any entry — when it was filed, what changed, and why — is readable, and every claim can be contested with the button on its page.