verifiable-instructions-such-length-format-constraints-strong-models CLM-0188

mechanismsingle paperReviewed by AI and Russ Miller

On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.

Capability: Following an unfamiliar procedure

Sources

supportsInstruction-Following Evaluation for Large Language Models
On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.

Reviewed by AI and Russ MillerLast checked: 2026-09-03Evidence activity

Either way, bring evidence: a link, or what you saw.

Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.

Related claims

When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.Following an unfamiliar procedure · reviewed by AI
On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
Models endorse widely held falsehoods, showing weak verification against what they know.Checking claims against evidence
Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.Fixing its own mistakes
Models complied with insecure completions a large fraction of the time, and more capable models were more likely to suggest insecure code.Writing secure code and dependencies