arxiv-2608-26867 SRC-0444 · paper

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

Created: 2026-08-27 · Ingested: 2026-10-17

Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestrati

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn a suite of verifier-checked tool-using ReAct task families (arithmetic, SQL analytics, security judgment, URL grounding, dependency planning, orchestration, tool policy), a single task-weighted aggregate accuracy hides inverted per-family rankings: the model that ranks last overall can lead both URL-grounding families while scoring lowest on structured SQL and multi-step tool families, which spread the four tested models by more than sixty points.