two-agent-embodied-social-tasks-requiring-coordinated-gaze-movement-expression CLM-0652In two-agent embodied social tasks requiring coordinated gaze, movement and expression, fine-tuning open VLMs (8B-14B InternVL) by timestep-level action imitation on demonstrations from an architecture with an explicit belief module shifts trial outcome scores up or down depending on condition but does not make the student's nonverbal signals more directionally coherent, because cross-channel communicative coordination is a trial-level property that per-timestep supervision does not encode.
Capability: Turning plans into physical action
Observed on
Open-source VLMs up to 15B in a single two-box guide-or-mislead scenario, 200 trials per model; no human baseline, no RL comparison..
Sources
- Validated by: Compared fine-tuned InternVL-8B/14B against their base models on ToM Outcome Conformance Score and on trajectory, gaze, expressivity and signal-sensitivity metrics across four conditions; cooperative outcomes declined, 14B trajectory alignment turned negative, signal sensitivity stayed negative throughout.. Measured on two fine-tuned InternVL variants against their own base models in one scenario; the attribution to timestep-level supervision is argued rather than isolated against a trial-level-reward arm, which the paper leaves unrun.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In two-agent embodied social tasks where one agent must guide or mislead another through movement, gaze and facial expression, open-source VLMs up to 15B show no reliable behavioral change when an explicit theory-of-mind order constraint is imposed, failing at two separate points: most do not produce directionally coherent nonverbal signals, and those that do (e.g. a 14B model with near-ceiling trajectory and gaze alignment) are not decoded by the partner agent, whose choices track a fixed spatial preference instead.Turning plans into physical action · reviewed by AI
- When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Turning plans into physical action · reviewed by AI
- For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Turning plans into physical action · reviewed by AI
- In embodied environments with high-frequency action spaces (Atari, Minecraft, Crafter, DMLab, Overcooked, Pico Park, MindCraft), pairing a frozen instruction-tuned VLM that emits sparse text instructions with a small world-model controller trained to follow them scores higher than either the same VLM emitting low-level actions directly or the same controller running without instructions, and the gain depends on the instructions carrying semantic content — random-string instructions score no better than no instructions at all.Turning plans into physical action · reviewed by AI
- In embodied multi-agent household planning with a fixed high-level skill interface, a team's task success rate overstates coordination quality, because goal predicates can be reached through illegal shortcuts: several of 11 tested MLLMs succeed on more episodes than they produce legal plans on (Claude Haiku 4.5 63.1% success vs 47.2% legal; Qwen3.6-Plus 69.1% vs 56.7%), while others honour constraints yet miss the goal.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A VLM fine-tuned purely on per-timestep action demonstrations that reaches high trajectory and gaze alignment and condition-appropriate signal sensitivity would refute it. Drafted stance toward two-agent-embodied-social-tasks-where-one-agent-must-guide: supports -- The fine-tuning result shows the two documented bottlenecks (incoherent signal production, failed decoding) persist after action-level imitation.