fixed-7b-code-backbone-short-python-tasks-mbpp-task-aware CLM-0570observationsingle paperReviewed by AI
For a fixed 7B code backbone on short Python tasks (MBPP), task-aware prompt routing plus LoRA fine-tuning on prompts rewritten into the routed style raises pass@1 only slightly (+1.1 points) when one sample is returned, but yields large gains (+9-10 points) once several candidates are sampled and selected by execution against the task's visible public tests.
Capability: Generating and editing working code
Observed on
Qwen2.5-Coder-7B-Instruct on MBPP sanitized and full, where visible public tests ship with each task; partial ablation only (no Verify-alone cell)..
Sources
- Validated by: pass@1 on MBPP: base 0.8276/0.7528 (sanitized/full) vs full pipeline 0.8911/0.8520; Route+Align without verification only 0.8366/0.7640; repeated 120-task subsets gave a 95% CI of [0.0250,0.1417] on Full and [-0.0194,0.0722] on Sanitized; Jaccard contamination check found zero overlap.. Measured on one benchmark, one backbone, no Verify-only cell, so the attribution to component interaction rests on comparing three combinations; the Sanitized confidence interval crosses zero under repeated subset evaluation; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.Improving its own scaffolding from failures · reviewed by AI
- When fine-tuning a 4B code model, training on auxiliary targets that name each program region's correctness obligations (derived from aligning an AST implementation graph with Lean proof-flow traces) raises solved counts over code-only SFT, but the auxiliary-target stage alone shifts the model off the Python-only output distribution and must be followed by a code-only consolidation stage to avoid losing broad function-level ability.Generating and editing working code · reviewed by AI
- In multi-agent LLM workflows with a mixed-size model zoo, choosing which model instantiates each agent step-by-step from an online estimate of task-progress gain plus remaining time and cost budget keeps task quality near the best single-model policy at lower energy than one-shot query-level routing or stall-triggered escalation.Getting more from several models than from one · reviewed by AI
- In multi-agent LLM pipelines over reasoning and code tasks with a pool of 3B-32B open-weight backbones, conditioning each per-step routing decision on a learned filtered memory rather than the full accumulated trajectory leaves accuracy about the same while cutting inference cost by roughly 40-50%, so most of the accumulated history is cost without accuracy value.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation isolating Verify alone that matches full-pipeline pass@1, or a setting where routing plus aligned LoRA gives most of the gain without multi-candidate execution selection.