real-world-online-rl-post-training-large-pretrained-vision-language-action-policies-high-precision CLM-0217For real-world online RL post-training of large pretrained vision-language-action policies on high-precision manipulation, optimizing the action head toward a paired advantage margin against a frozen reference action, rather than maximizing absolute Q values, yields higher autonomous success on long multi-step tasks, because absolute-Q maximization amplifies early optimistic value errors into policy drift that behavior-cloning regularization alone does not stop.
Capability: Turning plans into physical action
Observed on
Real-robot online RL with human intervention, large flow-based VLA backbones (pi-0.5 class) with only LoRA action-expert parameters trained, on millimeter-precision chemistry manip.
Sources
- Nine real tasks, four embodiments, 20 held-out trials per task: 98.3% mean success vs 67.8% for fine-tuned pi-0.5 and 10.0% for the strongest absolute-Q RL baseline (Robo-Dopamine), plus training curves on five tasks showing baselines stuck near 0-10% autonomous success. The comparison changes backbone, algorithm, and system at once, so the relative-advantage mechanism is argued from baseline failure modes rather than isolated by a matched ablation in the text provided.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For manipulation policies whose large VLA backbone cannot run within the control period, letting the slow model propose action chunks from a stale observation while a small RL-trained edit policy adjusts and Q-selects among those chunks using the latest observation recovers more performance than applying RL on top of asynchronous imitation-learning schemes alone, and the gap widens as inference delay or environment speed grows.Turning plans into physical action · reviewed by AI
- When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Turning plans into physical action · reviewed by AI
- For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Turning plans into physical action · reviewed by AI
- When RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Getting more from several models than from one · reviewed by AI
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation in which absolute-Q maximization with the same frozen-reference regularization, experience mix, and compute matches or exceeds relative-advantage updates in autonomous success on the same long-horizon precision tasks. Proposed technique, not catalogued: Reference-regularized relative-advantage policy improvement for VLA online RL.