On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.

Capability: Following an unfamiliar procedure

Sources

supportsInstruction-Following Evaluation for Large Language Models
On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.

Status: activeLast checked: 2026-09-03Evidence activity

Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.

Related claims

On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · unreviewed
Models endorse widely held falsehoods, showing weak verification against what they know.Checking claims against evidence
Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.Fixing its own mistakes
Models complied with insecure completions a large fraction of the time, and more capable models were more likely to suggest insecure code.Writing secure code and dependencies
On code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.Stating false facts confidently · unreviewed