masked-diffusion-language-models-llada-dream-asked-generate-around CLM-0210In masked diffusion language models (LLaDA, Dream) asked to generate around a frozen constraint span placed mid-output, failures to honor the constraint come partly from too little attention being allocated to that span during denoising: adding a fixed additive bias to attention logits on the span raises constraint adherence and overall quality up to a moderate bias, then degrades fluency and contextual fit as the bias grows.
Capability: Following an unfamiliar procedure
Observed on
Text-only in-place infilling tasks with a single anchor span; probed on two 7-8B masked diffusion models with a zero-parameter inference-time hook; behaviour under multiple or conf.
Sources
- Argued from a grid-searched scalar bias applied via forward pre-hooks, with 'anchor mass' (attention probability on constraint tokens) shown to rise with the bias and rubric-based GPT-5/JudgeLM scores tracing a non-monotonic curve. Only two models probed, no numbers for the peak reported in the text, and the judge protocol is itself LLM-based, so the quality drop at high bias is measured by a judge rather than by an independent fluency metric. The probe shows attention can be shifted and that it matters, but does not rule out other causes of failure.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Open-weight instruction-tuned models, told in plain language to not think about a concept while transcribing a sentence, suppress that concept's residual-stream projection enough to evade linear probes, the Jacobian lens, activation oracles and natural-language autoencoders, with no fine-tuning or prompt search — detection at a 1% false-positive budget falls to roughly 5-21% under suppression while engagement is caught near ceiling.Strategic deception and detecting it · reviewed by AI
- In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Grounding answers in the image · reviewed by AI
- On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.Losing information in long inputs · reviewed by AI
- In LVLMs asked to describe people (LLaVA, InstructBLIP, Shikra), masking attention so that text tokens read visual tokens but language-only self-attention is suppressed in late decoder layers lowers gender misattribution while keeping answer coverage high, whereas visual-only or language-only masking scores well on the bias metric only because the model names a gender in under 10% of cases.Grounding answers in the image · reviewed by AI
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Sweeping the attention-bias strength on the constraint span produces no region where constraint-following scores rise above the unbiased baseline, or the scores rise monotonically with bias rather than peaking and then falling. Proposed technique, not catalogued: attention-bias probe on the constraint span.