fine-tuning-small-multimodal-model-synthesized-constraint-rich-instructions-raises-adherenceFine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following an unfamiliar procedure
Observed on
Qwen3.5-4B target model fine-tuned on ~15k synthesized samples, evaluated on MM-IFEval C-Level vs P-Level splits.
Sources
- One ablation table on one benchmark with one 4B backbone: static pipeline moved C-Level from 62.7 to 64.9 while P-Level fell from 55.0 to 46.0, and adding reflection plus memory recovered P-Level to 54.0. Single seed, no variance reported; P-Level has only 100 items, so the trade-off size is loosely estimated.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · unreviewed
- When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Following an unfamiliar procedure · unreviewed
- When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.Following an unfamiliar procedure · unreviewed
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · unreviewed
- In 7-8B vision-language models, fine-tuning visual and text embeddings toward perturbation-averaged and ground-truth anchors before cross-modal contrastive alignment lowers object and attribute hallucination rates on captioning and existence benchmarks without reducing object coverage or general VQA accuracy.Grounding answers in the image · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A static constraint-rich synthesis pipeline that raises output-constraint accuracy without any drop on perception-grounded instruction items, across multiple target models. Proposed technique, not catalogued: verifier-bound self-evolving instruction synthesis.