tool-using-agent-trained-outcome-only-rl-surface-prompt-cue-co-occurs CLM-0498When a tool-using agent is trained with outcome-only RL and a surface prompt cue co-occurs with one tool in most training examples, the agent invokes that tool on tasks where it is useless, but only for tools it has already learned to use well and mainly when the cue is semantically related to the tool.
Capability: Using the tools it is given
Observed on
Qwen2.5-7B-Instruct trained with GRPO on a 1,000-example synthetic mix of Natural Questions (search) and DeepMath (Python), with injected cues; effect appears at 80%+ cue-task corr.
Sources
- Validated by: Measured as the cue-present minus cue-absent search-call rate on DeepMath and GSM8K against the same RL run without the dense reward: spurious deltas of +9.5 to +39.2 points fell to ~0, with task accuracy equal or higher than the no-cue baseline.. Controlled synthetic study, one base model, one RL algorithm, six cues, with counterfactual cue-present/absent test pairs and a swapped-cue control that separates competence from semantic alignment. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An RL run where a strongly correlated, semantically aligned cue produces large spurious invocation of a tool the agent never learned to use on its intended task, or where a semantically mismatched cue produces equally large spurious invocation of a well-learned tool.