scientific-tasks-where-domain-specific-constraint-general-presentation-constraint-letterOn scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following an unfamiliar procedure
Observed on
2,527 constraint-injected samples built from 13 existing scientific datasets across chemistry, geography, biology, materials science and physics; four closed-source and seven open-.
Sources
- Measured with per-constraint DRFR split by constraint domain and group: the reported gap for the top closed model is roughly 89% on scientific versus 75% on general constraints, and letter constraints sit near 51%. Constraints were injected into existing problems, so difficulty of the two constraint types is not matched — the gap could partly reflect that general constraints here are harder in absolute terms, not that they are neglected.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.Following an unfamiliar procedure · unreviewed
- When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.Following an unfamiliar procedure · unreviewed
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
- When a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.Generating and editing working code · unreviewed
- When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Following an unfamiliar procedure · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A benchmark of the same kind where models score equal or higher on general formatting constraints than on injected discipline-specific constraints, across model families. Drafted stance toward verifiable-instructions-such-length-format-constraints-strong-models: supports -- The paper reports that adherence falls as constraint counts rise and that general format/letter constraints are the weakest group even for the strongest model.