arxiv-2609-05324 SRC-0210 · paperRoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang
Created: 2026-09-04 · Ingested: 2026-09-19
https://arxiv.org/abs/2609.05324(opens in a new tab)Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and procedural complexity. We introduce \textbf{RoboSPA} (\textbf{Robo}t \textbf{S}patial-\textbf{P}rocedural \textbf{A}ssessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models. \texttt{RoboSPA} focuses on two core dimensions, Fine-Grained Spatial Reasonin
In brief
Vision-language-action models that clear standard manipulation benchmarks degrade once spatial relations get ambiguous and plans get long, and the failures separate into distinct causes rather than a single drop in success rate.
RoboSPA is a manipulation dataset and benchmark built around 2 axes, fine-grained spatial reasoning and long-horizon procedural planning, spanning 10 task categories and 56 base tasks. Each task is instantiated at 5 difficulty levels for 280 variants of rising spatial ambiguity and procedural complexity, with 527K collected trajectories across multiple embodiments and scenes. Beyond binary success, the benchmark adds diagnostic metrics intended to localise where a rollout breaks.
Evaluations on representative VLA models are reported as struggling with complex spatial relations, precise low-level execution, and memory-intensive planning. The abstract gives no per-model or per-level numbers, so the size of the degradation across the 5 difficulty levels is not quantifiable from what is available here.
The design does isolate difficulty, since each base task appears at all 5 levels, which is a within-task comparison rather than a cross-benchmark one. But the set of models evaluated is described only as representative, no baselines or ablations are named, and the diagnostic metrics are the authors' own.
Treat this as a source of graded manipulation tasks for stress-testing spatial and planning failures, not yet as a measured ranking of VLA systems.
Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsOn language-conditioned manipulation tasks where the same base task is instantiated at increasing levels of spatial ambiguity and procedural length, current VLA models degrade on fine-grained spatial relations, precise low-level execution, and memory-dependent multi-step planning, so success measured only on predefined simple settings overstates their competence.
- supportsWhen language-conditioned VLA manipulation tasks are instantiated at graded difficulty levels that add spatial ambiguity (fine-grained spatial relations, precise placement) and procedural length (multi-stage plans requiring memory of completed steps), representative VLA models degrade on both axes, so success on predefined single-stage settings does not indicate spatial or long-horizon competence.