multi-turn-agentic-coding-sessions-constraints-written-into-repository-policy CLM-0213For multi-turn agentic coding sessions, constraints written into a repository policy file (e.g. CLAUDE.md) hold up better over successive turns than the same kinds of constraints stated in user turns, and constraints that replace an earlier constraint are followed less often than newly added ones.
Capability: Following an unfamiliar procedure
Observed on
100 synthesized multi-turn software development instances (avg 7.04 turns, ~13 constraints per turn), 11 frontier code agents under the Claude Code harness, checklist verification .
Sources
- Measured as average constraint success rate split by constraint source across turns, averaged over all tested models; the gap is reported as a figure trend rather than with per-model statistics, and constraint content is not matched item-for-item between sources, so the source effect is not fully isolated from difficulty.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For coding-agent robot manipulation where a safety constraint (do not touch a named obstacle) is given only as text in the task prompt, the agent restates the constraint early and then plans the shortest path through the obstacle, and enforcing the same constraint with an external stateless verifier that re-checks each route against grounded obstacle boxes before motion raises both task success and collision avoidance.Turning plans into physical action · reviewed by AI
- For long-horizon repository-level coding agents, a small (4B) critic trained to diagnose each just-completed step with a severity label, error type and causal rationale, and to redirect the next action when severity is critical, repairs more initially-failing tasks with a single branch (14.1% fail-to-pass) than five full resamples of the base agent (13.7%), at about one-fifth the base-agent output tokens.Judging another's work as well as a careful reviewer would · reviewed by AI
- For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Improving its own scaffolding from failures · reviewed by AI
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
- When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.Following an unfamiliar procedure · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A comparison on similar multi-turn coding tasks where policy-file constraints decay as fast as, or faster than, user-instruction constraints, or where replacement constraints are followed at least as often as newly added ones.