arxiv-2608-29352 · paper

Cross-Relational Preference Learning for Better LLM Instruction Following

Created: 2026-08-29 · Ingested: 2026-09-09

Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Per

In brief

Preference data built by perturbing instructions so that the perturbed response space contains, partially overlaps, or is disjoint from the original one trains better instruction followers than perturbation schemes that only use the disjoint (opposing-constraint) case. The gain comes from the data construction, not the optimizer.

CRPL decomposes an instruction into atomic constraints with DeepSeek v3.1, generates perturbed variants targeting each of the 3 relationship types, samples positives and negatives from each sub-region of the combined response space, and verifies each atomic constraint with a generated code function that must exceed 80% accuracy on six LLM-generated test cases. Backbones are Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct; evaluation on IFEval, IFBench, a 200-instruction Perturbed-IF set, and FollowBench.

With DPO on Qwen2.5-7B, average accuracy rises from 52.8 (base) and 55.5 (AutoIF) to 62.4; on Llama3.1-8B, 65.2 versus 60.7 for AutoIF. Training size is matched at 12k pairs across all offline baselines. Ablations remove each relationship type (90.7 to 82.9-85.9 Acc_ins) and each sampling region, and compare verification schemes: atomic verification gives 0 false positives versus 4 for LLM-as-judge.

Analysis is confined to 1 backbone with IFEval training. RLVR baselines use different base models and 22k data, so that comparison is not matched. Online DPO gains are smaller and uneven.

If you build preference data by perturbing constraints, vary the relationship, not just the polarity.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.