vlm-driven-embodied-task-planning-egocentric-camera-where-hazards-fall CLM-0231For VLM-driven embodied task planning with an egocentric camera, where hazards can fall out of view, the form of the memory handed to a risk predictor decides whether safety improves: an object-relation graph that retains out-of-view objects and updates edges after actions raises safe success rate well above no memory, while feeding the full image history or a flat list of seen objects scores below having no memory at all, because relationships and structure, not object recall, are what risk prediction needs.
Capability: Turning plans into physical action
Observed on
GPT-4o planner and risk predictor, IS-Bench household tasks modified to a single egocentric RGB-D view plus a 4-task real Franka setup; hazards must have been observed at some earl.
Sources
- Measured on IS-Bench across four hazard categories (roughly 6 tasks per category, so small samples) and 10 real-robot runs per task. Average SSR 0.59 for graph vs 0.21 without graph memory, 0.13 full history, 0.38 RAG, 0.17 object list; removing invisible-object retention drops SSR to 0.21 and removing action-aware edge update to 0.38. Task success rate barely moves, so the effect is specific to safety. Single model, small per-cell counts, no seeds or variance reported.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For memory-augmented VLA manipulation policies that must actively move the viewpoint to find task-relevant evidence, writing one memory frame at each fixed action-chunk boundary generalizes better than event-triggered writing (on predicted subtask change or on gripper/head/torso motion), and extra memory capacity helps only under the regular schedule — event-triggered policies degrade further out of distribution because mistimed triggers omit informative views and control jitter fills memory with redundant frames.Turning plans into physical action · reviewed by AI
- In VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.Planning a sequence of actions before acting · reviewed by AI
- For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.Turning plans into physical action · reviewed by AI
- For multi-step assembly task planning from an action history and a workstation image, prompting a multimodal LLM to emit a structured (JSON) state representation and propagate it through each predicted action — then truncating the plan at the predicted state closest to the inferred goal — raises action executability and lowers final-state error relative to predicting actions alone, at the cost of shorter plans (lower overlap with recorded human action sequences) and roughly eight times the latency.Planning a sequence of actions before acting · reviewed by AI
- When VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.Checking claims against evidence · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation on the same benchmark in which full-history image context or a seen-object list matches the relation graph's safe success rate, or in which removing edges/out-of-view retention leaves safe success unchanged. Proposed technique, not catalogued: Object-relation graph memory with risk-predictor replanning.