vlm-based-navigation-agents-having-model-output-2d-pixel-egocentric CLM-0191For VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.
Capability: Turning plans into physical action
Observed on
Qwen2.5-VL-7B fine-tuned on 90k VLN-CE trajectories, indoor continuous navigation (R2R-CE, RxR-CE), depth available from simulator ground truth or a depth camera; degrades under se.
Sources
- One SFT-only ablation block holding architecture, data, sensing and controller fixed: swapping pixel-to-3D for a NavFoM-style metric waypoint output lowers success rate by 24.8 points on R2R-CE and 25.2 on RxR-CE. Single paper, single base model, authors' own reimplementation of the baseline output head; the mechanism (2D pretraining priors) is argued, not directly isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.Turning plans into physical action · reviewed by AI
- For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.Turning plans into physical action · reviewed by AI
- For VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.Turning plans into physical action · reviewed by AI
- In VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.Planning a sequence of actions before acting · reviewed by AI
- For pretrained 2D vision-language-action manipulation policies, adding 3D geometry only on the perception side is not enough: conditioning the action denoiser directly on a raw 3D semantic point field — at the specific diffusion blocks where linear probes show end-effector-coordinate error rising — raises bimanual task success beyond perception-side 3D fusion alone, and encoding the point cloud with a PointNet++-style encoder instead of keeping raw points lowers success.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-budget comparison in which the same base VLM, data, and controller achieve equal or higher success rate with direct metric-waypoint output than with pixel-pointing, or in which the pixel-pointing advantage disappears once depth is imperfect. Proposed technique, not catalogued: Pixel-pointing action space with depth projection.