building-preference-data-teach-7-8b-instruct-models-follow-multi-constraintWhen building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following an unfamiliar procedure
Observed on
Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, DPO/KTO/online-DPO, 12k pairs, constraints that are checkable by executable code; tested on IFEval, IFBench, a perturbed set built by.
Sources
- Two backbones, three optimization methods, four test sets, fixed 12k training size, with ablations removing each relationship type and each pair region — each removal lowers accuracy. One of the four test sets (Perturbed-IF) is generated by the authors' own perturbation method, which favors the method; the data-scaling curve and the fixed budget do argue the gain is not just more data. Single group, no independent replication.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · unreviewed
- Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.Following an unfamiliar procedure · unreviewed
- When preference pairs are built by having one strong model write all chosen responses and one weak model all rejected responses (delta learning), contrastive objectives such as DPO transfer the chosen model's sycophantic-agreement rate to the student even though no individual example contains sycophancy — the student's rate tracks the log-ratio of the two teachers' sycophancy rates, so reversing which teacher is 'chosen' drives sycophancy far below the SFT starting point.Telling the user what they want to hear · unreviewed
- When fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Stating false facts confidently · unreviewed
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Matching the same 12k pair budget with disjoint-only or random perturbations yields the same constraint accuracy, or the advantage disappears on a held-out benchmark not built with the same perturbation procedure. Proposed technique, not catalogued: Cross-relational preference pairs over perturbed constraints.