arxiv-2608-19598 SRC-0101 · paper

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

Created: 2026-08-20 · Ingested: 2026-09-15

https://arxiv.org/abs/2608.19598

Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address the

In brief

Standard DPO applied to multimodal models leaves them unable to tell an image apart from the same image with its critical visual evidence removed, and adding an image-level preference term fixes much of it. The authors call this visual insensitivity and split it into across-image and within-image forms, backed by a margin decomposition showing both correspond to near-zero margins that the new loss directly enlarges.

mPEA-DPO builds rejected images by masking 9% of each image, repeated n = 30 times, then picking the variant with lowest CLIP similarity to the original. Training used LLaVA-v1.5-7B and 13B on 22K preference instances over 13K images, evaluated on MMHalBench, Object HalBench, and AMBER.

On the 7B model, MMHalBench score rises from 2.11 to 3.02 and hallucination from 0.54 to 0.36; Object HalBench CHAIRs drops 53.6 to 4.3. The 13B model reaches 3.16 score and 0.31 hallucination. Recall stays at 52.30 versus DAMA's 54.50, arguing the gain is not just shorter answers.

Evidence includes component ablations, an alpha sweep, a data-scale curve, and 4 alternative rejected-image strategies (blockwise, rotation, crop, blackness), all worse. Only 1 model family was tested, no recent MLLMs, and coverage on AMBER falls from 51.0 to 46.7.

If you train multimodal preference models, the rejected-image construction matters more than the loss form.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen adding image-level preference pairs to multimodal DPO for LLaVA-1.5 7B/13B, constructing the rejected image by selecting, among many equal-size random masks, the variant with lowest CLIP similarity to the original lowers object-hallucination rates more than coarse alternatives (blockwise masking, rotation, cropping, or a fully blacked-out image), at the cost of slightly lower object coverage.