procedure-following · activeFollowing an unfamiliar procedure
Models drift from a new multi-step procedure or set of constraints back to familiar routines.
Also called: instruction following, constraint adherence, workflow compliance
Tags: agentic, autonomous-agent, chat-assistant, coding-agent, customer-support
When a task comes with explicit instructions, constraints, or a step-by-step workflow the model has not seen in training, a capable model follows it exactly, honors every constraint at once, and does not substitute a familiar but different routine.
Claims
- observationsingle paperWhen building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.
- observationsingle paperFine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.
- mechanismmechanism reasoningA correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
- mechanismsingle paperDecomposing an instruction into explicit yes/no items and judging against them beats asking for a holistic score, and the same checklist turned on the model's own draft improves the draft. The gain comes from making each requirement separately checkable, not from the model trying harder — so it is largest on instructions carrying several constraints at once.
- mechanismsingle paperMany failures come from not following domain policy documents that were provided in the prompt.
- mechanismsingle paperOn realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.
- observationsingle papercontestedGiving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
- observationsingle paperOn scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.
- mechanismsingle paperOn verifiable instructions such as length and format constraints, strong models still fail a meaningful share, and failures grow when several constraints apply at once.
- observationsingle paperWhen a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.
Techniques
- Keep a guide file of past failuresprocessA versioned instruction file the agent reads before acting, where every line is a past failure converted into a permanent rule.
- Turn the procedure into a checklistprocessConvert constraints and steps into an explicit checklist the model must tick off before finishing.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.