arxiv-2608-23758 SRC-0113 · paper

EXAM$^2$: $\underline{Ex}tending$ $\underline{A}udio$ $Understanding$ $in$ $\underline{M}ultilingual$ $and$ $\underline{M}ultimodal$ $Analysis$

Created: 2026-08-24 · Ingested: 2026-09-15

Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain largely constrained to English and narrow audio domains. Prior benchmarks typically focus on a single audio modality, i.e., speech, sound, or music, limiting the systematic investigation into how these models generalize across diverse visual scenarios. In this paper, we introduce EXAM$^2$, a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, including speech, sound, music, mixed-audio settings, and v

In brief

Adding a generated image for each answer choice raises audio question-answering accuracy, but only when fusion is native to the model; cascaded pipelines that convert audio to text first get worse when images are added.

EXAM2 restructures MMAU-test-mini, MMAR, Clotho and AudioMCQ-Clotho into 5,667 multiple-choice questions over speech, sound, music and mixed audio, with 22,614 generated images (Stable Diffusion, GPT-image-2) and 135,684 translated instances in English, German, Spanish, Japanese, Malay and Chinese. Models tested include GPT-4o-audio, GPT-5-mini, GPT-5-2, Phi-4-multimodal-instruct, Qwen-2.5-omni and Gemma3n-E4B, plus a LoRA fine-tune (Gemma3n-EXAM2) trained with uniform language sampling.

GPT-4o-audio leads at 73.83% average. Gemma3n-EXAM2 reaches 61.24%, best open-source overall, but loses on sound to cascaded Gemma3n (59.26% vs 67.12%). Qwen-2.5-omni gains +15.55% from adding images to audio; cascaded Gemma3n drops when images are added. Western languages outscore Japanese and Chinese throughout.

The modality and language claims are backed by ablations on the same fine-tuned model: multimodal 61.22% vs audio-only 60.05% vs image-only 41.23%, and English-only training loses 12.4 points against 6-language training, with English itself gaining 3.4 points. Paired t-tests are reported (p < 0.05, p < 0.01).

Caveats: images are synthetic and machine-generated, translations come from GPT-5-mini, the tuning result is 1 model family, and no matched-compute baseline for the fine-tune is reported.

If you build audio pipelines, this argues against late fusion and for multilingual training even when you only care about English.

Written from the abstract by claude-opus-5 on 2026-09-22, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn multiple-choice audio questions where each answer choice is paired with a generated image, adding the images helps native multimodal models most on environmental-sound questions and not at all on speech (speech is best audio-only), and in cascaded pipelines that convert audio to transcripts or captions first the images lower average accuracy.