treating-post-deployment-improvement-scoped-behavioural-maintenance-where-each-recurring CLM-0336observationsingle paperReviewed by AI
Treating post-deployment improvement as scoped behavioural maintenance, where each recurring failure becomes a skill patch with a stated scope and injection point that is promoted only after passing its targeted cases, a protected regression suite and negative controls, raised correctness from 3.70 to 4.55 on an SEC-filing QA benchmark, and over twelve operational rounds promoted only 6 of 33 candidate skills while the monitoring non-correct rate fell from 20.0% to 12.5%; the ungated condition scored higher on each candidate's own targeted cases but admitted regressions on the protected set.
Capability: Improving its own scaffolding from failures
Observed on
SEC filing question answering, 2026; evolution, protected and monitoring sets held separate; 33 candidate skills over 12 rounds; developer-reported, no public code stated..
Sources
- Validated by: Correctness on an enhanced SEC-filing benchmark before and after evolved skills, a 12-round operational study reporting promotion counts and monitoring non-correct rate, and a gated-versus-ungated comparison showing the ungated variant wins on targeted cases while regressing protected ones. One domain.. The gated-versus-ungated contrast is the transferable part and it replicates the gate finding from EvoOntology and SoL-Pi in a third setting: acceptance criteria, not the improvement engine, are what make a loop ratchet rather than drift. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.Writing secure code and dependencies · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- For image-borne injected instructions delivered to tool-using agents, completion rate understates exposure: agents act on the injected instruction about ten times more often than they complete the objective (12.8% attempted vs ~1% completed across 720 runs), and most failures stop at the planning step where the model read the payload and simply did not act on it rather than at any architectural barrier.Following instructions hidden in data · reviewed by AI
- For tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.Following instructions hidden in data · reviewed by AI
- When injecting new or counterfactual facts into a 3-4B instruct model by post-training, supervised fine-tuning on extracted QA pairs binds facts to their training question form — recall of the single fact stays reasonable while multi-source and inferential questions over the same facts collapse — and adding more SFT epochs does not close the gap, whereas mixed-policy RL that supplies the golden answer as an off-policy trajectory when on-policy rollouts fail recovers much of the generalization.Absorbing and flexibly using new knowledge · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A domain where the protected suite cannot be kept representative, so gated promotion blocks real gains without preventing regressions, or a gated-versus-ungated comparison over a longer horizon where the ungated arm ends ahead.