vlms-choosing-ordered-route-over-numbered-waypoints-single-egocentric CLM-0508observationsingle paperReviewed by AI
For VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.
Capability: Turning plans into physical action
Observed on
Zero-shot, single first-person RGB image with numbered waypoints and JSON output; feasibility scored against registered 3D scene geometry; indoor simulated scenes..
Sources
- supportsEgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language ModelsMeasured on 1,111 benchmark questions across nine VLMs, with route diagnostics decomposed into output validity, endpoint choice, first-edge legality and full-route legality; length- and waypoint-density stratification reported. Single benchmark, authors' own construction; human reference reaches 54.2 EgoPath Score versus 28.6 for the best VLM.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For routing a video-production request to an ordered chain of registered sub-agents, an 8B open model fine-tuned on descriptor-conditioned routing plans (SFT on verified plans plus GRPO rewarding validity, sub-agent coverage F1 and dependency-consistent order) picks chains more accurately than much larger prompted planners, with the gap widest on long, structurally unfamiliar chains.Getting more from several models than from one · reviewed by AI
- For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.Turning plans into physical action · reviewed by AI
- For VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.Turning plans into physical action · reviewed by AI
- In multi-hop KGQA evaluated as graph navigation with annotated gold evidence paths, reported agreement between an agent's traversed path and the reference path cannot be credited to question conditioning unless it is compared against question-agnostic references — unbiased random walks and answer-oracle shortest paths — because dense or small graphs let structure alone deliver both answer reachability and partial path overlap.Whether the measurement made the finding · reviewed by AI
- For VLM agents doing multi-phase navigation and pick-and-place in procedurally generated indoor scenes with moving human obstacles, subtask progress runs well ahead of full success — agents reach the first goal and then stall or loop after the phase transition — so success rate alone hides where the failure is.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model whose first-edge legality and full-route legality are close together, or whose embodied-route legality approaches its point-route legality, on the same benchmark.