embodied-environments-high-frequency-action-spaces-atari-minecraft-crafter-dmlab CLM-0380In embodied environments with high-frequency action spaces (Atari, Minecraft, Crafter, DMLab, Overcooked, Pico Park, MindCraft), pairing a frozen instruction-tuned VLM that emits sparse text instructions with a small world-model controller trained to follow them scores higher than either the same VLM emitting low-level actions directly or the same controller running without instructions, and the gain depends on the instructions carrying semantic content — random-string instructions score no better than no instructions at all.
Capability: Turning plans into physical action
Observed on
Controllers are trained per environment (~800M params) with behaviour cloning on VLM-relabelled segments of their own rollouts; matched observation and action spaces across compare.
Sources
- Validated by: Official per-environment metrics on seven environments against controller-only, direct VLM action generation, a fine-tuned VLA (RT-2), DreamerV3, MAPPO and QMIX; ablation over instruction source (VLM-generated 11.1, template 9.9, clustered 8.9, random strings 8.3, none 8.2, environment-derived 11.4 on Minecraft Diamond); instruction-following accuracy 86-97% across planners.. Measured across seven environments with four VLM planners and internal ablations; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For frozen VLA policies executing repetitive, perceptually aliased manipulation tasks (counting pick-and-place), a progress memory that advances after each attempted action converts transient grasp or placement failures into persistent task-state errors, so writes must be gated on observed physical evidence of the intended transition; gating this way raised success far above the frozen policy and above memory-augmented baselines.Turning plans into physical action · reviewed by AI
- When adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.Turning plans into physical action · reviewed by AI
- For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Turning plans into physical action · reviewed by AI
- When a multimodal LLM acts as the decision core of a simulated humanoid reacting to sudden household hazards, picking the correct reaction primitive does not mean the reaction is executable: even on correctly classified scenes, hands land a median 0.48 m from the true impact point, 89% of misses fall short, and in 49% of those decisions the body never steps forward and stands beyond arm's reach.Turning plans into physical action · reviewed by AI
- Open-weight instruction-tuned models, told in plain language to not think about a concept while transcribing a sentence, suppress that concept's residual-stream projection enough to evade linear probes, the Jacobian lens, activation oracles and natural-language autoencoders, with no fine-tuning or prompt search — detection at a 1% false-positive budget falls to roughly 5-21% under suppression while engagement is caught near ceiling.Strategic deception and detecting it · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Under matched observation and action spaces, a controller-only or direct-VLM-action baseline matching or beating the planner-plus-controller system on these environments, or random-string instructions producing gains comparable to semantically meaningful ones.