fine-tuning-small-multimodal-model-synthesized-constraint-rich-instructions-raises-adherence CLM-0051Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.
Capability: Following an unfamiliar procedure
Observed on
Qwen3.5-4B target model fine-tuned on ~15k synthesized samples, evaluated on MM-IFEval C-Level vs P-Level splits.
Sources
- One ablation table on one benchmark with one 4B backbone: static pipeline moved C-Level from 62.7 to 64.9 while P-Level fell from 55.0 to 46.0, and adding reflection plus memory recovered P-Level to 54.0. Single seed, no variance reported; P-Level has only 100 items, so the trade-off size is loosely estimated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
- When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Following an unfamiliar procedure · reviewed by AI
- When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.Following an unfamiliar procedure · reviewed by AI
- In embodied environments with high-frequency action spaces (Atari, Minecraft, Crafter, DMLab, Overcooked, Pico Park, MindCraft), pairing a frozen instruction-tuned VLM that emits sparse text instructions with a small world-model controller trained to follow them scores higher than either the same VLM emitting low-level actions directly or the same controller running without instructions, and the gain depends on the instructions carrying semantic content — random-string instructions score no better than no instructions at all.Turning plans into physical action · reviewed by AI
- On language-conditioned manipulation tasks where the same base task is instantiated at increasing levels of spatial ambiguity and procedural length, current VLA models degrade on fine-grained spatial relations, precise low-level execution, and memory-dependent multi-step planning, so success measured only on predefined simple settings overstates their competence.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A static constraint-rich synthesis pipeline that raises output-constraint accuracy without any drop on perception-grounded instruction items, across multiple target models. Proposed technique, not catalogued: verifier-bound self-evolving instruction synthesis.