paired-solvable-unsolvable-math-problems-translated-into-french-greek CLM-0465On paired solvable/unsolvable math problems translated into French and Greek, natively multilingual models (Gemma-4-31B, Qwen3-4B/30B) encode solvability most sharply in English yet agree least with that internal belief in their stated verdict in English, being more faithful in the lower-resource languages, while a weakly multilingual model (Llama-3.1-8B) shows the opposite ordering.
Capability: Saying what actually drove the answer
Observed on
Six instruction-tuned models, English/French/Greek translations of ReliableMath (313 solvable, 1,102 unsolvable), faithfulness defined as agreement between a pooled linear probe on.
Sources
- Measured on one benchmark (ReliableMath) machine-translated and human-validated into two languages; faithfulness uses ground-truth solvability as a proxy for internal belief and an LLM judge (Llama-3.3-70B, validated on 100 samples) for the verdict, so the probe may partly measure task label rather than belief. Differences between languages are a few points (e.g. Qwen3-4B 0.691 En vs 0.728 Fr/Gr); no confidence intervals given in the excerpt for Table 2.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In a two-agent extract-then-answer pipeline on multilingual extractive QA, routing the inter-agent handoff through English (forward-translate facts, answer in English, back-translate) lowers Exact Match relative to keeping handoffs in the user's language, and the loss grows with distance from English (13.0 points Spanish up to 30.6 points Hindi with Aya-23-8B).Getting more from several models than from one · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- When instruction-tuned models (Qwen2.5 7B-32B) make function calls from non-English user requests, they often select the right tool and argument names but emit argument values in the wrong language, a failure that exact-match or AST tool-calling metrics record as a generic invocation error rather than a language mismatch.Using the tools it is given · reviewed by AI
- On 1,010 counterfactual edits rooted in Chinese cultural entities and aligned with English counterparts, an edit made in one language largely does not propagate to the other: edits succeed in the edited language (about 80% for Chinese at batch size 10, falling with batch size) while the other language keeps answering the old fact, so multilingual models store the same fact in language-specific ways that editing does not bridge.Absorbing and flexibly using new knowledge · reviewed by AI
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Measuring probe-vs-verdict agreement on more models and languages and finding faithfulness highest in the dominant pretraining language for multilingual models too, or showing the reported gap disappears once probe accuracy differences across languages are accounted for.