arxiv-2608-28058 · paper

Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models

Created: 2026-08-28 · Ingested: 2026-09-09

Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input. Existing mitigation methods mainly rely on external supervision, output calibration, or attention regulation, leaving the internal representation dynamics of autoregressive generation underexplored. We identify an inference-time failure mode in which cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction and increasing hallucination risk. We propose \emph{Dynamic Alignment Com

In brief

Hallucinated tokens in vision-language models carry a measurable signature in hidden states: large Jensen-Shannon divergence between adjacent decoder layers and large drift across decoding steps, both concentrated in middle-to-late layers. Damping those two divergences at inference reduces hallucination without training.

Dynamic Alignment Compensation (DAC) adds a residual from the previous layer when inter-layer JS exceeds a threshold (LSC) and pulls the current-token state back toward a cached previous-step state when temporal JS exceeds a threshold (SSC). It is evaluated on 9 benchmarks (POPE, CHAIR, HallusionBench, MME, MM-Vet, MMBench, MMBench-CN, TextVQA, LLaVA-Bench) across LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL-3B/7B, Qwen3-VL-4B, LLaVA-NeXT-7B.

On LLaVA-1.5, CHAIR Cs/Ci falls from 47.6/13.3 to 28.2/7.3, POPE average accuracy rises 79.83 to 87.32, HallusionBench qACC 8.13 to 17.05. Per-token latency goes from 36.8 to 37.1 ms.

The component ablation on MM-Vet separates LSC (33.3) from SSC (33.0) against a 31.1 baseline, and dataset-level JS diagnostics on CHAIR back the mechanism. Recall drops on CHAIR (80.6 to 72.0) and output length shortens (99.7 to 88.9); the length-normalized check is deferred to an appendix. Comparisons come from 1 group, no independent replication.

Untested: closed-source models, video, multi-image, long-context. Thresholds are fixed, not tuned per dataset.

If you run open-source LVLMs, this is a near-free decoding change worth trying before contrastive decoding methods that cost 2x latency.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsIn open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.