prompt-imposes-k-deterministically-verifiable-constraints-once-per-constraint-pass CLM-0627When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.
Capability: Following an unfamiliar procedure
Observed on
Pure conjunctive composition of 36 rule-verifiable lexical, structural, format, numeric and relational constraints; 15 models across 8 families; k=1-12; raw generation with no scaf.
Sources
- Measured: 369,753 deterministic constraint checks, 4,527 probes, 15 models. Aggregate mCSR fits 72.0%x0.922^(k-1) with 0.2pp held-out MAE on k=9-12; at k=8 per-constraint 40.7% vs 5.7% all-pass. Mean pairwise phi = +0.067 over 601 pairs, only 1 negative. Difficulty, order and parameter confounds controlled. Does not cover sequential or conditional composition, or semantic constraints.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.Following an unfamiliar procedure
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
- When a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.Generating and editing working code · reviewed by AI
- When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.Following an unfamiliar procedure · reviewed by AI
- For multi-turn agentic coding sessions, constraints written into a repository policy file (e.g. CLAUDE.md) hold up better over successive turns than the same kinds of constraints stated in user turns, and constraints that replace an earlier constraint are followed less often than newly added ones.Following an unfamiliar procedure · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: high. Falsifier as drafted: Finding substantial negative pairwise co-failure correlations, or a selection/ordering of constraints that keeps all-pass rates high at k=8-12, or a model whose all-pass rate stays far above the product of its per-constraint rates. Drafted stance toward verifiable-instructions-such-length-format-constraints-strong-models: supports -- It quantifies the failure growth as constraints accumulate and attributes it to multiplicative accumulation rather than interaction.