arxiv-2605-01203 SRC-0354 · paper

GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models

Created: 2026-05-02 · Ingested: 2026-09-19

Currently, process reward models (PRMs) have exhibited remarkable potential for test-time scaling. Since large language models (LLMs) regularly generate flawed intermediate reasoning steps when tackling a broad spectrum of reasoning and decision-making tasks, PRMs are required to possess capabilities for detecting process-level errors in real-world scenarios. However, existing benchmarks primarily focus on mathematical reasoning, thereby failing to comprehensively evaluate the error detection ability of PRMs across diverse reasoning scenarios. To mitigate this gap, we introduce GR-Ben, a proce

In brief

Process reward models trained on math do not transfer their step-level error detection to scientific or logical reasoning, and prompted general LLMs beat them there by a wide margin.

LSR-Ben contains 3,595 human-annotated problem-solution pairs across 9 subdomains (physics, chemistry, biology, computer science; deductive, inductive, abductive, analogical, mixed logic), with solutions generated by 11 open-source and 4 closed-source LLMs, 13.7 average steps, and labelled first-error index plus error type. 13 PRMs and 11 LLMs were scored by F1 over accuracy on correct and erroneous samples, with ProcessBench as a math reference.

PRMs average under 30 on LSR-Ben versus 73.5-78.3 on ProcessBench for the strongest ones; best PRM is Qwen2.5-Math-PRM-72B at 37.4 average. Gemini-3.8-flash reaches 63.8, DeepSeek-v3.2 55.4. PRMs average 52.7 FN and 22.7 FP on flawed processes; LLMs 26.6 FN and 28.6 FP, so PRMs miss errors and LLMs over-flag them. Slow thinking raises every model tested (Qwen3-32B 36.7 to 45.4) except analogical reasoning.

Measured, not argued, with 3-annotator agreement (76.4% unanimous) and scalar-score PRMs thresholded on a held-out dev set. Non-math PRMs number only 2, so the math-training attribution rests on a small sample, and no PRM was retrained on these domains.

If you rely on a PRM outside math, expect it to approve broken steps.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOutside mathematics, the step-level error detection of both process reward models and general LLMs is markedly weaker than on math benchmarks, and the two fail differently: PRMs miss knowledge-based errors while LLMs miss computational errors, across science and logic domains.