arxiv-2609-07160 SRC-0216 · paper

In-Place Instruction Following in Diffusion Language Models

Zheng Nie, Zherui Li, Jiaming Zhang, Kun Wang, Zhenhong Zhou, Yufei Guo

Created: 2026-09-07 · Ingested: 2026-09-19

Diffusion Large Language Models (dLLMs) generate text via bidirectional iterative denoising, naturally supporting user-specified constraints anchored at arbitrary output positions, a paradigm known as In-place Prompting (IPP). We formalize this as the In-place Instruction Following (IIF) task and construct IIF-Bench, a hierarchical benchmark spanning literal, style, and discourse-function constraints, paired with a rubric-based local-global evaluation protocol. An inference-time attention-bias probe suggests that vanilla dLLMs often under-prioritize constraint spans during denoising. We then p

In brief

Diffusion language models can anchor a fixed span at an arbitrary output position and generate around it, but off-the-shelf they under-weight that span during denoising, and this is a training problem rather than a capability gap.

The authors formalize In-place Instruction Following and build IIF-Bench, roughly 5K instances over 3 levels: literal phrase preservation, 12 literary styles, and 8 discourse functions, scored on 4 rubric dimensions by GPT-5 and JudgeLM. Graft adds constraint-aware SFT (loss masked off the anchor) plus DPO on preference pairs balancing local adherence and global consistency, tested on LLaDA-1.5, LLaDA-8B, Dream-7B and MMaDA against vanilla versions and prompt-based AR models.

Average overall IIF score rises from 57.75 to 73.10, with gains of 15.91 on literal and 15.57 on discourse constraints. GPT-4o still leads at L3 (70.42 vs 69.53 for LLaDA-8B+Graft). An attention-bias probe shows moderate bias helps and excessive bias hurts fluency, supporting the prioritization account.

Effects are measured with SFT-only versus full-pipeline ablations, frequency-penalty and tag-suppression ablations, and 2 judges averaged over 3 runs; human evaluation is deferred to an appendix, judging is LLM-based, and no equal-compute AR fine-tuning baseline exists.

Text-only, English, 4 mid-size dLLMs; behavior under conflicting anchors or at larger scale is untested. Treat in-place control as trainable, not native.

Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsIn masked diffusion language models (LLaDA, Dream) asked to generate around a frozen constraint span placed mid-output, failures to honor the constraint come partly from too little attention being allocated to that span during denoising: adding a fixed additive bias to attention logits on the span raises constraint adherence and overall quality up to a moderate bias, then degrades fluency and contextual fit as the bias grows.