frozen-weight-video-agent-open-ended-surgical-video-qa-mining-candidate CLM-0574For a frozen-weight video agent on open-ended surgical video QA, mining candidate retrieval skills from the agent's own low-scoring traces and keeping each only when it raises a four-dimensional validation reward on ~40 held-out questions beats installing a hand-written general 'looking discipline' skill, and the single distilled temporal skill (re-read the window adjacent to a localized event for before/after questions) carries most of the gain.
Capability: Improving its own scaffolding from failures
Observed on
Text-only orchestrator with frozen vision sub-agents, long (18-28 min median) surgical videos, open-ended free-form answers judged by an LLM rubric validated against neurosurgeon r.
Sources
- Validated by: Component ablation on the 259-question neurosurgery split: full harness averaged 2.90 across four 1-5 judge dimensions; removing directed re-look -0.38, removing coarse-to-fine -0.16, replacing distilled skills with a hand-written looking-discipline skill -0.13. Judge agreement with experts kappa 0.72 (neuro) and 0.86 (SVU).. Measured on one benchmark the authors built (259 neurosurgery + 864 SVU questions), with ablations only on the neurosurgery split. Gains over the matched one-shot backbone are +0.17 to +0.56 per dimension. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For self-evolving LLM agents that admit a new skill into a persistent skill bank whenever aggregate outcome reward rises, the accepted skills are collectively net-negative and the gate cannot do better in principle, because the bank-level gain is one scalar that leaves every incumbent skill's contribution unconstrained; scoring sampled coalition knockouts at the gate, and masking transferred skills whose removal improves retrieval quality on unlabeled target queries, recovers the missing composition and domain dependence.Improving its own scaffolding from failures · reviewed by AI
- When an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.Improving its own scaffolding from failures · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- For 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.Reasoning about time in video · reviewed by AI
- For long-video QA with a frozen VLM and fixed answering interface, an outer loop that proposes, executes and retains only development-accuracy-improving rewrites of the executable context-construction program can advance past a strong hand-crafted keyframe-selection harness, and the gain survives a matched cumulative visual-token control and direct reuse on other long-video benchmarks.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison in which a hand-written or ungated skill library reaches the same or better four-dimensional average as the reward-gated distilled library, or in which removing directed re-look costs no more than removing any other component. Drafted stance toward self-evolving-agents-accumulate-reusable-skills-bank-admit-new-skill: supports -- The paper argues that without an automatic success signal self-evolving loops accumulate plausible-but-useless skills, and adds a per-candidate reward gate with a per-question regression check to prevent that.