arxiv-2601-17397 SRC-0344 · paper

CLM-Bench: Benchmarking and Analyzing Cross-lingual Misalignment of LLMs in Knowledge Editing

Created: 2026-01-24 · Ingested: 2026-09-19

Knowledge Editing (KE) has emerged as a promising paradigm for updating facts in Large Language Models (LLMs) without retraining. However, progress in Multilingual Knowledge Editing (MKE) is currently hindered by biased evaluation frameworks. We observe that existing MKE benchmarks are typically constructed by mechanically translating English-centric datasets into target languages (e.g., English-to-Chinese). This approach introduces translation artifacts and neglects culturally specific entities native to the target language, failing to reflect the true knowledge distribution of LLMs. To addre

In brief

Knowledge edits applied in Chinese do not propagate to English, and the parameter updates for the two languages sit in near-orthogonal subspaces. Edits made in both languages at once behave as the linear sum of the monolingual updates rather than as a shared interlingual representation.

CLM-Bench is a Chinese-first CounterFact set of 1,010 entries across 24 knowledge domains, generated with Deepseek-R1 and manually reviewed, then aligned to English counterparts instead of being translated from English. Batch editing with MEMIT, AlphaEdit, and PMET (via EasyEdit) was run on Llama3-8B, Mistral-7B, Qwen2-7B, and llama2-chinese-7B under a lifelong sequential protocol, 1000 edits per monolingual pass and 2000 bilingual triplets for mixed editing.

For Llama3-8B with MEMIT at batch 1000, Chinese editing reached zh_score 61.71% and zh Reliability 63.21% while en Reliability was 19.55%. Geometric analysis of layer 12 mlp_down_proj deltas gave cosine similarity 0.027–0.042 between Chinese and English edits, neuron Jaccard below 0.32, subspace angles above 60 degrees; for mixed vs the sum, cosine above 0.976, relative error below 0.23, subspace angle falling from 63 degrees at batch 1 to 12 degrees at batch 1000.

The effect is measured, with batch-size sweeps (1, 10, 100, 1000) and layer ablations that leave the pattern intact, but there is no independent replication and no non-locate-and-edit baseline. AlphaEdit and PMET sometimes collapse below 20% at scale, confounding their numbers. Only Chinese-English, batch locate-and-edit methods, and a single analysed layer were tested.

If you were counting on edits transferring across languages, plan to edit each language separately.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn 1,010 counterfactual edits rooted in Chinese cultural entities and aligned with English counterparts, an edit made in one language largely does not propagate to the other: edits succeed in the edited language (about 80% for Chinese at batch size 10, falling with batch size) while the other language keeps answering the old fact, so multilingual models store the same fact in language-specific ways that editing does not bridge.