simulated-urban-air-ground-object-search-where-vlm-drives-both CLM-0220In simulated urban air-ground object search where a VLM drives both a UAV and a UGV, wrapping the model in a fixed search-handoff-verify protocol that leaves geometric projection and road-graph path planning to deterministic tools raises task success for backbones with adequate perception while cutting decision calls for all, but does not lift weak small open models, whose success can fall on harder splits.
Capability: Turning plans into physical action
Observed on
210 held-out CARLA-Air episodes in one unseen town, three difficulty levels, nine VLM backbones, 80 high-level decision budget, idealized communication and known map; static vehicl.
Sources
- supportsTowards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic MethodPaired baseline-vs-protocol runs on 210 episodes: success rate improved for 8 of 9 backbones, decision steps fell for all 9, mean +22.5 points and -33.3 steps; Gemini-3.6-Flash hard split 8.6 to 55.7 SR. Qwen3-VL-4B and Qwen3-VL-30B-A3B dropped on mid/hard. The design compares the whole protocol against plain prompting and does not ablate individual tools, so which component carries the effect is not isolated; single test town.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For training-free MLLM-driven navigation (instruction-following and object-goal) in simulated indoor environments, gating each planner-proposed action on evidential support, geometric reachability and subgoal consistency before dispatch — and refusing a stop request unless semantic, geometric and progress checks all pass — raises success rate over letting the planner act directly, but the stop check buys completion at some cost in path efficiency.Turning plans into physical action · reviewed by AI
- For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.Turning plans into physical action · reviewed by AI
- In VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.Planning a sequence of actions before acting · reviewed by AI
- For VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.Turning plans into physical action · reviewed by AI
- For VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A paired comparison on the same benchmark where the protocol gives no success gain for strong backbones, or where weak small models gain as much as strong ones. Proposed technique, not catalogued: role-conditioned search-handoff-verify protocol with deterministic projection and path planning tools.