tool-augmented-legal-agents-multi-step-chinese-law-tasks-correct-final-answer CLM-0496observationsingle paperReviewed by AI
For tool-augmented legal agents on multi-step Chinese-law tasks, a correct final answer does not indicate a clean reasoning trajectory: across 18 agent configurations, trajectories that reach the right answer still contain at least one substantive legal hallucination about 68% of the time on average, and that rate barely moves with backbone scale or agentic framework (0.61-0.78), while procedural-process hallucination rates vary widely (0.07-0.85).
Capability: Stating false facts confidently
Observed on
Chinese (PRC) law, 3,414 expert-curated instances filtered to hard cases, objective-question subset for the right-answer analysis, rubric-anchored LLM-as-a-judge over full trajecto.
Sources
- Validated by: Not measured against human error-labelling agreement in the given text; validated indirectly by showing the metrics separate frameworks and scale monotonically within the LRAS family.. Measured on one benchmark with an LLM judge over 18 agent configurations; hard-case filtering (only items all six filter models failed) inflates absolute rates, and the judge's own reliability is only referenced to an appendix robustness analysis not included here.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.Whether the measurement made the finding · reviewed by AI
- Given a published state-of-the-art method as the starting problem, a fully autonomous multi-agent loop (extract limitations, generate and novelty-filter ideas, run experiments, select the best hypothesis, plan and run ablations, draft the paper, pass a simulated peer-review rebuttal) improved on the human baseline for 86 of 107 top-venue papers (80%) by an average 25% relative, and its papers scored above the mean of accepted ICLR 2026 and NeurIPS 2025 papers under automated AI reviewers; every judgement in that result, from the improvement parsed out of result tables to the review scores, was made by a model, not a person.Improving its own scaffolding from failures · reviewed by AI
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- For claim-level hallucination detection on long-document QA, estimating a claim's reliability from how central it is in a graph of sampled model responses cannot separate true claims from confidently repeated false ones, because both get high agreement; adding trusted source documents as anchor nodes with higher-weight edges shifts the basis from internal consensus to external grounding and yields a Grounded pool with few hallucinations (97.6% precision on QASPER), at the cost of conservatively leaving correct-but-novel claims on the boundary.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An independent step-level audit (human expert) of correct-answer legal-agent trajectories finding substantive hallucination in a small minority of cases, or showing the judge's flags are mostly false positives.