arxiv-2602-20065 SRC-0539 · paper

Multilingual Large Language Models do not comprehend all natural languages to equal degrees

N. Moskvina, Raquel Montero, Masaya Yoshida, F. Hubers, Paolo Morosi, Walid Irhaymi

Created: 2026-02-23 · Ingested: 2026-09-23

https://arxiv.org/abs/2602.20065

Large Language Models (LLMs) play a critical role in how humans access information. While their core functionality relies on the ability to comprehend written requests, our understanding of this ability is currently constrained, because most benchmarks evaluate LLMs in high-resource languages that are predominantly spoken by Western, Educated, Industrialized, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less natural or reliable outputs, even in multilingual, state

In brief

English is not the best language for LLM comprehension. Across a comprehension task, several Romance languages systematically outperformed English, including lower-resource ones.

The setup: 3 popular LLMs prompted on a single language comprehension task in 12 languages spanning the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic families, with human baselines for comparison.

The reported results are qualitative in the source. Models scored high across typologically diverse languages but fell short of human baselines in every language, with the size of the gap varying by language. The English-versus-Romance ordering is described as systematic but no accuracy figures are given here.

Evidence is measured against human baselines rather than argued, and the cross-linguistic comparison is held on 1 task. Explanatory factors such as tokenization, distance from Spanish and English, training data size, and high- versus low-resource or WEIRD versus non-WEIRD data origin are discussed, not isolated by ablation, so the cause of the English deficit is not attributed.

Boundaries: 3 models, 12 languages, 1 comprehension task, no generation or reasoning tasks, and no reported per-language numbers in the abstract.

If you default to English prompts on the assumption they are strongest, that assumption deserves testing on your own task.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.