manipulation-policies-pretrained-mixed-egocentric-human-robot-data-adding-egocentric CLM-0436mechanismsingle paperReviewed by AI
For manipulation policies pretrained on mixed egocentric-human and robot data, adding egocentric data helps only when the embodiment gap is bridged explicitly: a three-stage curriculum (ego priors, then task-matched human/robot alignment in a shared end-effector frame, then robot-only) raised real-robot success, while separate ego/robot action heads gave only a marginal in-distribution gain and shared future-video prediction in a world-action model made success worse than a robot-only baseline.
Capability: Turning plans into physical action
Observed on
Pretraining on ~2,659 hours of ego+robot data; pi0.5-style VLA and Fast-WAM architectures; identical post-training and evaluation on seven dual-arm tasks, 10 trials per task per co.
Sources
- Validated by: Real-robot multi-task evaluation: combined success 53.57% vs 32.86% for the robot-only VLA baseline (ID 64.29% vs 42.86%, OOD 42.86% vs 22.86%), 140 trials per checkpoint, matched steps and hyperparameters.. Measured on one dual-arm platform, seven tasks, 10 ID + 10 OOD trials each, so per-task counts are noisy and several tasks score 0/10 for all methods; one seed, one corpus, and the ordering of representation-similarity metrics did not match the success ordering.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Turning plans into physical action · reviewed by AI
- For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Turning plans into physical action · reviewed by AI
- For deciding when a robot skill has finished and the next subtask should begin, general-purpose VLMs prompted zero-shot are near useless on long-horizon mobile manipulation (under 4% of transitions detected on BEHAVIOR-1K, including a frontier proprietary model), while a sub-1B student fine-tuned on compact self-explanations condensed from teacher reasoning over annotated demonstrations detects about 90% and lifts closed-loop task success from 0.30 to 0.53 against a 0.63 oracle.Turning plans into physical action · reviewed by AI
- For LLM-driven robot manipulation where every tool call is schema-validated and every motion kinematically and collision-checked before execution, execution-time recovery mechanisms (retrying a failed plan with an alternative candidate, resetting to a collision-checked safe pose between stages) contribute more to end-to-end task success than canonical action sequencing, because most residual failures are transient planning or IK failures rather than wrong orderings.Turning plans into physical action · reviewed by AI
- In VLM-driven human-aware robot task planning over scene graphs, the bottleneck is inferring and grounding the symbolic goal, not the planning: once both human and robot goals correctly cover the ground truth, a formal multi-agent planner produces a valid, conflict-free joint plan in 99.6% of trials, while end-to-end success with VLM-inferred human goals is 69.7%.Planning a sequence of actions before acting · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A controlled pretraining run at matched data and compute where plain ego-robot mixing (or shared future-video co-training) matches or beats the alignment curriculum on real-robot success.