arxiv-2409-12558 SRC-0316 · paper

RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues

Created: 2024-09-19 · Ingested: 2026-09-19

In real-world applications with Large Language Models (LLMs), external retrieval mechanisms - such as Search-Augmented Generation (SAG), tool utilization, and Retrieval-Augmented Generation (RAG) - are often employed to enhance the quality of augmented generations in dialogues. These approaches often come with multi-turn dialogue, where each interaction is enriched by relevant information retrieved from external sources. Existing benchmarks either assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in single-turn settings. However, there is a g

In brief

Giving a model the right retrieved context does not stop it degrading as a multi-turn dialogue adds conditions; performance drops turn over turn even when the context is correct.

RAD-Bench builds 89 multi-turn samples, each 3 turns with retrieved context and a reference answer, for 267 evaluated turns across 6 scenarios split into Retrieval Synthesis (academic writing, news TLDR, education) and Retrieval Reasoning (finance, customer support, travel planning). Questions are synthetically generated by LLMs, filtered by LLM scorers plus manual inspection, and responses are scored 1 to 10 by a reference-guided GPT-4o judge.

4 closed and 8 open models were tested. GPT-4o leads with 8.72 average, then GPT-4o-mini 8.47, Mistral-Large 7.91, Llama3.1-405B 7.88, Deepseek-v2 7.86, GPT-3.5-Turbo 6.59. Travel planning is hardest (Mistral-Nemo-12B 3.55; Deepseek-v2 7.95 beats GPT-4o's 7.83). Retrieval Reasoning scores fall each turn; Synthesis rises at turn 2 then falls at turn 3.

The degradation is measured but not causally isolated: no ablation separating context quality, turn position, and task difficulty, and the turn-3 synthesis dip is explained post hoc from judge transcripts. Scoring is single-judge GPT-4o, which shares a family with the top scorers, and BreeXe-8x7B also served as a question scorer, a bias the authors flag.

Use it to pick models for retrieval-augmented chat, not to estimate absolute quality.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen accurate retrieved context is supplied at every turn of a three-turn dialogue, model scores still fall as each turn adds a condition or constraint on how the retrieved material must be used, and models that score alike on plain multi-turn chat diverge on retrieval-grounded turns, so using retrieval well is a distinct competence from chatting well.