monocular-vlm-based-navigation-policies-making-model-first-emit-two-token CLM-0444For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.
Capability: Turning plans into physical action
Observed on
Monocular RGB navigation with a pretrained VLM backbone (Qwen3-VL-4B), staged embodied-reasoning mid-training, SFT with DAgger and GRPO post-training; simulation benchmarks plus qu.
Sources
- Validated by: Reported state-of-the-art monocular success rates across 10 public navigation simulation settings (R2R/RxR val-unseen, MP3D and HM3D v1/v2 ObjectNav, HM3D-OVON, EVT-Bench STT/DT) with one checkpoint and no benchmark-specific fine-tuning; the tokenizer's 0.72 cm average displacement error versus 2.48 cm for a single-level 4096-entry planning vocabulary; embodied-reasoning checkpoint scored on 8 spatial benchmarks. No ablation isolating the pointing prefix is shown in the given text.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.Turning plans into physical action · reviewed by AI
- For VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.Turning plans into physical action · reviewed by AI
- When VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.Turning plans into physical action · reviewed by AI
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
- For VLM-driven embodied task planning with an egocentric camera, where hazards can fall out of view, the form of the memory handed to a risk predictor decides whether safety improves: an object-relation graph that retains out-of-view objects and updates edges after actions raises safe success rate well above no memory, while feeding the full image history or a flat list of seen objects scores below having no memory at all, because relationships and structure, not object recall, are what risk prediction needs.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation removing the pointing prefix (action tokens only) matching or beating the full model on the same navigation settings, or the unified single-backbone model failing to match task-specific policies on any of the three task families.