arxiv-2609-04442 SRC-0205 · paperGRACE: Graph-Grounded Reflective Agent Copilot Engine for Expert-in-the-Loop Knowledge Expansion
John Seon Keun Yi, Joshua R. Minot, Dokyun Lee
Created: 2026-09-03 · Ingested: 2026-09-19
https://arxiv.org/abs/2609.04442(opens in a new tab)Large language models deployed in high-stakes settings frequently generate plausible but ungrounded claims. Standard retrieval-augmented generation (RAG) pipelines offer limited remedy, since they retrieve isolated passages without tracking cross-document evidence relationships or quantifying uncertainty. We introduce GRACE (Graph-grounded Reflective Agent Copilot Engine), a framework that deconstructs LLM responses into atomic claims and grounds them against trusted knowledge priors within a weighted bipartite graph. Edge weights encode the closeness of each claim to the priors, enabling weig
In brief
Anchoring claim-level uncertainty to external trusted documents, rather than to agreement across sampled LLM responses, separates grounded facts from confidently repeated hallucinations and yields calibrated abstention. GRACE decomposes responses and trusted priors into atomic claims in a weighted bipartite graph, uses weighted closeness centrality to label each claim Grounded, Refuted, or Boundary, and routes only high-priority uncertain Boundary claims to a verifier under a Return on Attention rule.
Evaluation covers QASPER (100 papers) and QuALITY (50 articles) with GPT-4o-mini and LLaMA 3.1 8B, against zero-shot partial/full context, RAG, and GraphRAG.
On QASPER GRACE reaches 28.1% accuracy on both models versus 19.8% and 18.5% for RAG, with answerability near accuracy (50.6%, 50.5%) while GraphRAG reports 90.4% and 95.2% answerability. Across 3 expansion rounds on 334 QASPER questions, QA accuracy rises 14.1%→49.1%, above the full-paper context 33.6%; prior coverage 63.1%→83.4%, frontier 10.02→4.57 claims/paper, grounding precision 97.6%. Scaling from 1 to 50 documents (91→4,554 claims) costs 52.18%→47.16%.
The expansion loop is measured with an LLM verifier (GPT-4.1) standing in for experts; a 15-paper, 15-student human round matched coverage (93.63% vs 93.58%) and accuracy but resolved more frontier, with no gold key so Yield is acceptance not correctness. Only 1 expansion round was tested with humans, 2 models, and priors are assumed equally trustworthy.
If you build retrieval for audited settings, centrality over priors is a cheap calibration signal worth testing against GraphRAG.
Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsFor claim-level hallucination detection on long-document QA, estimating a claim's reliability from how central it is in a graph of sampled model responses cannot separate true claims from confidently repeated false ones, because both get high agreement; adding trusted source documents as anchor nodes with higher-weight edges shifts the basis from internal consensus to external grounding and yields a Grounded pool with few hallucinations (97.6% precision on QASPER), at the cost of conservatively leaving correct-but-novel claims on the boundary.
- supportsWhen claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.