Following an unfamiliar procedure

Claims

• observationsingle paper↑1↓0
  When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.
• mechanismsingle paper↑1↓0
  In a deployed multilingual medical triage system, most of the accuracy gain comes from writing the clinicians' implicit decision tree into an explicit scoped rule set rather than from moving the final decision out of the LLM: replacing a flat danger-sign list with 231 structured rules in the same end-to-end prompt raised recall from 0.565 to 0.837 and F1 from 0.606 to 0.689, while further handing the decision to a deterministic rule engine over LLM-extracted symptoms left accuracy statistically unchanged (recall 0.810, F1 0.702).
• observationsingle paper↑1↓0
  Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.
• mechanismmechanism reasoning↑2↓0
  A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
• mechanismsingle paper↑1↓0
  Decomposing an instruction into explicit yes/no items and judging against them beats asking for a holistic score, and the same checklist turned on the model's own draft improves the draft. The gain comes from making each requirement separately checkable, not from the model trying harder — so it is largest on instructions carrying several constraints at once.
• mechanismsingle paper↑1↓0
  Many failures come from not following domain policy documents that were provided in the prompt.
• mechanismsingle paper↑1↓0
  In masked diffusion language models (LLaDA, Dream) asked to generate around a frozen constraint span placed mid-output, failures to honor the constraint come partly from too little attention being allocated to that span during denoising: adding a fixed additive bias to attention logits on the span raises constraint adherence and overall quality up to a moderate bias, then degrades fluency and contextual fit as the bias grows.
• mechanismsingle paper↑1↓0
  When models must apply abstract, entity-independent procedural rules drawn from a shared pool, accuracy is limited by locating the right rules rather than by applying them: giving only the gold rules yields much higher exact-match than embedding those same rules in a pool of 100 or 1,000, and accuracy falls as the pool grows.
• observationsingle paper↑1↓0
  For multi-turn agentic coding sessions, constraints written into a repository policy file (e.g. CLAUDE.md) hold up better over successive turns than the same kinds of constraints stated in user turns, and constraints that replace an earlier constraint are followed less often than newly added ones.
• mechanismsingle paper↑1↓0
  When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.
• mechanismsingle paper↑1↓0
  On realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.
• observationsingle papercontested↑1↓2
  Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
• observationsingle paper↑1↓0
  On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.
• observationsingle paper↑1↓0
  For timeseries-encoder-fused 8B LLMs answering questions that require chaining several temporal primitives, supervised fine-tuning on worked decompositions only helps on composition families whose decompositions were demonstrated, while RLVR with outcome-verifiable rewards on top of that checkpoint also lifts composition families that received no demonstration (+20.1 pp) and held-out compositions (+9.6 pp).
• mechanismsingle paper↑1↓0
  On verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.
• observationsingle paper↑1↓0
  When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.

Techniques

• Externalize the expert decision tree into a clinician-editabprocess
  Restrict the LLM to extracting canonical symptoms and patient context from a clinician-curated vocabulary, and let a deterministic rule engine owned by clinicians make the emergency decision.
• Keep a guide file of past failuresprocess
  A versioned instruction file the agent reads before acting, where every line is a past failure converted into a permanent rule.
• Turn the procedure into a checklistprocess
  Convert constraints and steps into an explicit checklist the model must tick off before finishing.

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding2026-09-14
  ↑ supports a claim here
• TimeThink: Eliciting Compositional Reasoning in Timeseries Large Language Models2026-09-11
  ↑ supports a claim here
• Auditable Emergency Triage for Maternal and Newborn Care in India2026-09-08
  ↑ supports a claim here
• In-Place Instruction Following in Diffusion Language Models2026-09-07
  ↑ supports a claim here
• Cross-Relational Preference Learning for Better LLM Instruction Following2026-08-29
  ↑ supports a claim here
• VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following2026-08-26
  ↑ supports a claim here
• SciMIF: Understanding Multimodal Instruction Following in Scientific Domains2026-08-26
  ↑ supports a claim here
• Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios2026-08-26
  ↑ supports a claim here
• Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models2026-08-24
  ↑ supports a claim here
• Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction2026-08-12
  ↑ supports a claim here
• Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory2026-08-07
  about this capability · no claim drafted yet
• Harness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook2026-08-01 · post
  ↑ supports a claim here · ↓ contests a claim here
• Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?2026-02-12
  ↑ supports a claim here
• TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and GenerationThis source: TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and…How much the field cites it — steadily cited in the last 12 months33 in the last 12 months · 61 totalpublished 2024checked 2026-09-04click for Semantic Scholar2024-10-04
  ↑ supports a claim here
• My AI Adoption Journey – Mitchell Hashimoto2026 · post
  ↑ supports a claim here · ↓ contests a claim here
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsThis source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Dom…How much the field cites it — very heavily cited in the last 12 months889 in the last 12 months · 1075 totalpublished 2024checked 2026-09-042024
  ↑ supports a claim here
• WebArena: A Realistic Web Environment for Building Autonomous AgentsThis source: WebArena: A Realistic Web Environment for Building Autonomous AgentsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 1931 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch2023
  ↑ supports a claim here
• Instruction-Following Evaluation for Large Language ModelsThis source: Instruction-Following Evaluation for Large Language ModelsHow much the field cites it — very heavily cited in the last 12 months625 in the last 12 months · 1081 totalpublished 2023checked 2026-09-042023
  ↑ supports a claim here