vlms-must-operate-unfamiliar-household-devices-no-manual-through CLM-0509When VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.
Capability: Turning plans into physical action
Observed on
Documentation-free device operation on 968 rendered 3D household devices with annotated component bounding boxes, discrete atomic actions and fixed viewpoint switching; six VLMs, R.
Sources
- Measured on VGEBench: success rates 1.48-54.27% across six VLMs; perfect-step sub-task rate 12.94% vs 62.86% sub-task success for the best model; GPT-5-mini's successful-navigation rate was 12.5% below Gemini-3-Flash while its average coordinate distance to the nearest target box was 63.9% worse (77.3% worse weighted for repeated failures). Human baseline on 149 episodes reached 74.50% SR vs 56.38%. Renders, not real scenes; bottleneck attribution rests on comparing two differently-scaled metrics, not a controlled ablation.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.Turning plans into physical action · reviewed by AI
- When VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.Checking claims against evidence · reviewed by AI
- For VLA manipulation policies doing single-step action prediction on LIBERO observations, the commanded action is fragile to camera rotation but tolerant of brightness: validated over entire continuous perturbation regions, 27% of instances admit a counterexample within ±1° of camera rotation and 51% within ±2°, while every instance is robust to a ±2% brightness shift and the median tolerates ±30%.Turning plans into physical action · reviewed by AI
- For VLM agents doing multi-phase navigation and pick-and-place in procedurally generated indoor scenes with moving human obstacles, subtask progress runs well ahead of full success — agents reach the first goal and then stall or loop after the phase transition — so success rate alone hides where the failure is.Turning plans into physical action · reviewed by AI
- For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A benchmark of the same kind where model coordinate error on target components is near human level while view-switching success lags, or where perfect-step sub-task rates approach overall sub-task success rates, would show localization is not the bottleneck.