agentic-retrieval-augmented-multi-hop-qa-7b-class-policy-models-scoring-each CLM-0359mechanismsingle paperReviewed by AI
For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.
Capability: Carrying a chain of inference through to its conclusion
Observed on
Qwen2.5-7B-Instruct (plus Llama-3.1-8B, Qwen2.5-3B) policies, Wikipedia retrieval with k=3, five single- and multi-hop QA benchmarks; gains largest on 2WikiMultiHopQA and absent/ne.
Sources
- Validated by: EM/F1 on PopQA, HotpotQA, 2WikiMultiHopQA, Bamboogle, MuSiQue against Search-R1, ReasonRAG, StepSearch; best average (34.5 EM / 44.1 F1), ablation removing PRM scoring drops to 32.7/41.9; Bamboogle gap not significant (p=0.66); reranking study shows math/general PRMs degrade with larger N.. Measured on five QA benchmarks with one main backbone plus three generalization backbones; component ablation isolates the PRM's contribution (+1.8 EM average), and a reranking experiment shows math/general PRMs fall below majority voting as candidate pool grows. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When reward models are scored by sequence log-probability on contextual preference pairs, abstentions are systematically shorter than the unsupported answers they are preferred over, so measured grounding/abstention judgement partly tracks response length; rewriting rejected and refusal responses to match the preferred ones in length while keeping the grounding failure raises pairwise accuracy far more than adding more in-domain legal preference data (up to +25.6 points over baseline for Ministral-8B).Judging another's work as well as a careful reviewer would · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- When adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.Turning plans into physical action · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-data comparison where preference pairs built from final-answer-derived step scores match or beat grounding-aware step labels on multi-hop QA, or where the PRM ablation shows no drop.