scientific-tasks-where-domain-specific-constraint-general-presentation-constraint-letter CLM-0142On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.
Evidence for: Law of triviality (breaks)
Capability: Following an unfamiliar procedure
Observed on
2,527 constraint-injected samples built from 13 existing scientific datasets across chemistry, geography, biology, materials science and physics; four closed-source and seven open-.
Sources
- Measured with per-constraint DRFR split by constraint domain and group: the reported gap for the top closed model is roughly 89% on scientific versus 75% on general constraints, and letter constraints sit near 51%. Constraints were injected into existing problems, so difficulty of the two constraint types is not matched — the gap could partly reflect that general constraints here are harder in absolute terms, not that they are neglected.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.Following an unfamiliar procedure · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
- Open-weight instruction-tuned models, told in plain language to not think about a concept while transcribing a sentence, suppress that concept's residual-stream projection enough to evade linear probes, the Jacobian lens, activation oracles and natural-language autoencoders, with no fine-tuning or prompt search — detection at a 1% false-positive budget falls to roughly 5-21% under suppression while engagement is caught near ceiling.Strategic deception and detecting it · reviewed by AI
- When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.Following an unfamiliar procedure · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A benchmark of the same kind where models score equal or higher on general formatting constraints than on injected discipline-specific constraints, across model families. Drafted stance toward verifiable-instructions-such-length-format-constraints-strong-models: supports -- The paper reports that adherence falls as constraint counts rise and that general format/letter constraints are the weakest group even for the strongest model.