vlm-driven-human-aware-robot-task-planning-over-scene-graphs-bottleneck CLM-0507mechanismsingle paperReviewed by AI
In VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.
Capability: Planning a sequence of actions before acting
Observed on
Photorealistic household simulation (VirtualHome, 4 scenes), one human and one mobile robot, cooperative goal-consistent human assumed, PDDL-style domain with FMAP planner, GPT-5.6.
Sources
- Validated by: Joint success rate 69.71% vs 35.50% for VLM-as-planner and 34.42% for GG-LLM, with paired cluster bootstrap CIs on 48 scene-activity-task configurations; also higher robot and human goal completion and about half the functional conflict rate.. Measured in simulation with an oracle-goal ablation isolating grounding error (SR 69.71 to 96.38; 99.60% planner success on 1,490 correctly-grounded trials). Success drops to 25.5% on complex two-item activities, where goal prediction fails most. Single simulator, one planner, hand-written domain; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In embodied multi-agent household planning with a fixed high-level skill interface, a team's task success rate overstates coordination quality, because goal predicates can be reached through illegal shortcuts: several of 11 tested MLLMs succeed on more episodes than they produce legal plans on (Claude Haiku 4.5 63.1% success vs 47.2% legal; Qwen3.6-Plus 69.1% vs 56.7%), while others honour constraints yet miss the goal.Getting more from several models than from one · reviewed by AI
- For VLM agents doing multi-phase navigation and pick-and-place in procedurally generated indoor scenes with moving human obstacles, subtask progress runs well ahead of full success — agents reach the first goal and then stall or loop after the phase transition — so success rate alone hides where the failure is.Turning plans into physical action · reviewed by AI
- For deciding when a robot skill has finished and the next subtask should begin, general-purpose VLMs prompted zero-shot are near useless on long-horizon mobile manipulation (under 4% of transitions detected on BEHAVIOR-1K, including a frontier proprietary model), while a sub-1B student fine-tuned on compact self-explanations condensed from teacher reasoning over annotated demonstrations detects about 90% and lifts closed-loop task success from 0.30 to 0.53 against a 0.63 oracle.Turning plans into physical action · reviewed by AI
- For coding-agent robot manipulation where a safety constraint (do not touch a named obstacle) is given only as text in the task prompt, the agent restates the constraint early and then plans the shortest path through the obstacle, and enforcing the same constraint with an external stateless verifier that re-checks each route against grounded obstacle boxes before motion raises both task success and collision avoidance.Turning plans into physical action · reviewed by AI
- For multi-step assembly task planning from an action history and a workstation image, prompting a multimodal LLM to emit a structured (JSON) state representation and propagate it through each predicted action — then truncating the plan at the predicted state closest to the inferred goal — raises action executability and lowers final-state error relative to predicting actions alone, at the cost of shorter plans (lower overlap with recorded human action sequences) and roughly eight times the latency.Planning a sequence of actions before acting · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Substituting ground-truth human and robot goals leaves end-to-end success far below ceiling, or the symbolic planner fails on a large fraction of correctly grounded problems in comparable domains.