building-preference-data-teach-7-8b-instruct-models-follow-multi-constraint CLM-0027When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.
Capability: Following an unfamiliar procedure
Observed on
Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, DPO/KTO/online-DPO, 12k pairs, constraints that are checkable by executable code; tested on IFEval, IFBench, a perturbed set built by.
Sources
- Two backbones, three optimization methods, four test sets, fixed 12k training size, with ablations removing each relationship type and each pair region — each removal lowers accuracy. One of the four test sets (Perturbed-IF) is generated by the authors' own perturbation method, which favors the method; the data-scaling curve and the fixed budget do argue the gain is not just more data. Single group, no independent replication.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
- Fine-tuning a small multimodal model on synthesized constraint-rich instructions raises adherence to output-level constraints (format, word count, keywords) while lowering accuracy on perception-grounded instructions, so the average can fall below the base model unless the synthesis loop tracks per-constraint failures and image compatibility.Following an unfamiliar procedure · reviewed by AI
- When preference pairs are built by having one strong model write all chosen responses and one weak model all rejected responses (delta learning), contrastive objectives such as DPO transfer the chosen model's sycophantic-agreement rate to the student even though no individual example contains sycophancy — the student's rate tracks the log-ratio of the two teachers' sycophancy rates, so reversing which teacher is 'chosen' drives sycophancy far below the SFT starting point.Telling the user what they want to hear · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.Following an unfamiliar procedure · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Matching the same 12k pair budget with disjoint-only or random perturbations yields the same constraint accuracy, or the advantage disappears on a held-out benchmark not built with the same perturbation procedure. Proposed technique, not catalogued: Cross-relational preference pairs over perturbed constraints.