arxiv-2608-12282 SRC-0617 · paper

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

Created: 2026-08-12 · Ingested: 2026-10-24

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing pr

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsFor LLM agents under a fixed ReAct harness answering questions over executable database-backed APIs plus document retrieval, accuracy falls sharply as the required reasoning depth grows from 1 to 5 hops (most models lose over half their accuracy), and the surviving failures concentrate in language-mediated steps — entity disambiguation, cross-source grounding, schema alignment — rather than in emitting syntactically valid tool calls.