arxiv-2609-24210 SRC-0527 · paper

ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation

Created: 2026-09-21 · Ingested: 2026-10-17

Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward signal. Large Multimodal Models (LMMs) play a natural critical role in jointly assessing chart visual appearance and task requirements. They are therefore increasingly used as visual critics and reward models, yet their reliability as judges remains largely unexplored. To this end, we introduce ChartJudgeBench, a diagnostic vision-language benchmark for assessing LMM judges in chart-to-code workflows. It includes 1,003 Chart Perception Alignme

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen open-source vision-language models are post-trained with RL or critique tuning to act as judges of chart-to-code outputs, they become more likely to output Accept rather than better at verification: pairwise-comparison accuracy barely improves under an order-swapped protocol, while acceptance rates rise and specificity falls, so they ratify flawed but plausible-looking charts.