arxiv-2512-16698 SRC-0340 · paper

Do Multi-Agents Solve Better Than Single? Evaluating Agentic Frameworks for Diagram-Grounded Geometry Problem Solving and Reasoning

Created: 2025-12-18 · Ingested: 2026-09-19

Diagram-grounded geometry problem solving is a critical benchmark for multimodal large language models (MLLMs), yet the benefits of multi-agent design over single-agent remain unclear. We systematically compare single-agent and multi-agent pipelines on four visual math benchmarks: Geometry3K, MathVerse, OlympiadBench, and We-Math. For open-source models, multi-agent consistently improves performance. For example, Qwen-2.5-VL (7B) gains +6.8 points and Qwen-2.5-VL (32B) gains +3.3 on Geometry3K, and both Qwen-2.5-VL variants see further gains on OlympiadBench and We-Math. In contrast, the close

In brief

Splitting diagram-grounded geometry into an Interpreter agent (VLM emitting symbolic literals) and a Solver agent (LLM reasoning over them) helps weaker open-source models and mostly hurts a strong proprietary one.

The comparison covers Geometry3K (601 multiple-choice test problems), MathVerse (788-problem mini-test), OlympiadBench, and We-Math, with Qwen-2.5-VL 7B/32B (4-bit quantized) and Gemini-2.0-Flash as solvers; the Interpreter is fixed to Gemini-2.0-Flash. Zero-shot, frozen weights, Pass@3 scoring, LLM-as-a-judge (Gemini-2.0-Flash) for free-form answers.

Qwen-2.5-VL 7B gains +6.8 on Geometry3K and +9.4 on OlympiadBench; 32B gains +3.3 and +6.67. Gemini-2.0-Flash loses in multi-agent mode on Geometry3K (83.86% vs 85.19%), MathVerse (-0.45) and OlympiadBench (-2.46), but gains +1.74 on We-Math. Qwen 7B drops -6.0 on MathVerse.

An Interpreter ablation isolates the cause: swapping the Interpreter from Qwen-2.5 7B to Gemini moves Qwen-3 solver accuracy from 42.26% to 79.53%, and predicate-description cosine similarity tracks it (0.849 Gemini, 0.794 GPT-4o mini, 0.677 Qwen-2.5). Compute cost of the 2-stage pipeline is not matched against single-agent.

Only 2 model families, 4-bit quantization, 1 fixed prompt, no fine-tuning, and free-form judging uses the same model as the Interpreter.

Treat agentic decomposition as a fix for weak perception, not a general gain.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn four diagram-grounded geometry benchmarks a multi-agent pipeline consistently helps open-source vision-language models (Qwen-2.5-VL 7B +6.8 and 32B +3.3 points on Geometry3K, with further gains on OlympiadBench and We-Math) but a strong closed model (Gemini 2.0 Flash) generally did better as a single agent on the classic benchmarks, gaining only modestly on the newest one, so agentic decomposition pays for weaker models and less familiar tasks rather than universally.