multi-hop-kgqa-evaluated-graph-navigation-annotated-gold-evidence-paths CLM-0212In multi-hop KGQA evaluated as graph navigation with annotated gold evidence paths, reported agreement between an agent's traversed path and the reference path cannot be credited to question conditioning unless it is compared against question-agnostic references — unbiased random walks and answer-oracle shortest paths — because dense or small graphs let structure alone deliver both answer reachability and partial path overlap.
Capability: Whether the measurement made the finding
Observed on
Path-based KGQA/KG-completion agents scored with edit-distance or set-overlap path-fidelity metrics on a fixed shared navigation graph; matters most where out-degree is low or the .
Sources
- Argued from the paper's protocol design plus reported structural diagnostics: random-walk answer reachability is around 9% on Kinship versus roughly 0.6% on MQuAKE-ST, and graph density and clustering differ by orders of magnitude across the three graphs. The source text provided cuts off before the main results table, so no measured gap between agents and the calibration references is available here.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For routing a video-production request to an ordered chain of registered sub-agents, an 8B open model fine-tuned on descriptor-conditioned routing plans (SFT on verified plans plus GRPO rewarding validity, sub-agent coverage F1 and dependency-consistent order) picks chains more accurately than much larger prompted planners, with the gap widest on long, structurally unfamiliar chains.Getting more from several models than from one · reviewed by AI
- For long-horizon LLM agents on tool-use and terminal benchmarks, token-level confidence (predictive entropy, sequence probability) barely separates successful from failed trajectories, and the separable signal lives in the dependency structure of the trajectory: scoring a trajectory by propagating per-step uncertainty over a graph of reasoning, tool-call and environment-feedback nodes raises failure-detection AUROC over both local-confidence and sequence-based agent UQ baselines.Using the tools it is given · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- For VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.Turning plans into physical action · reviewed by AI
- For small open-weight models (1.5-4B) generating answers in a closed-domain RAG setup, hallucinated generations show a broken hidden-state proximity graph between the retrieved context entity and the generated entity — connectivity below 20% and path lengths above 7 hops, versus near-full connectivity at about 3 hops for grounded answers — giving a training-free detection signal (AUROC 0.89 on Qwen2.5-1.5B).Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that on these navigation datasets random-walk and shortest-path references score near zero on RED/PED/F1_Rel/F1_SG while trained agents score high, so the calibration baselines add no information about what drives path agreement. Proposed technique, not catalogued: Question-agnostic structural calibration baselines for path-fidelity metrics. Automatic check flagged: figures not in the source: 0.6%.