across-ten-languages-six-models-harmbench-advbench-unsafe-response-rates CLM-0289observationsingle paperReviewed by AI
Across ten languages and six models on HarmBench and AdvBench, unsafe-response rates for the same model differ by more than 50 points between languages; high-resource languages are safer on plain harmful queries but more vulnerable to adversarial-prompt jailbreaks, and simple defenses work but their effect depends on language and model, so safety measured in English does not transfer.
Capability: Refusing harmful requests under obfuscation
Observed on
Ten high- to low-resource languages, six LLMs, two jailbreak families (logical-expression and adversarial-prompt), 2025; unsafe rate judged automatically..
Sources
- Automatic harm judging; the cross-language spread is large enough to survive judge noise. Complements the multilingual prompt-injection finding filed under multilingual-disparity.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across six frontier chat models asked to produce phishing, deceptive sites and malware through direct and multi-stage prompts, non-English prompts were complied with more often than English ones, with a mean full-compliance rate of about 76% across natural languages (English about 71%, some languages above 97% on some models), so safety alignment is thinner outside English; simple encodings (Base64, ROT13, hex) lowered compliance to about 59% but did not remove it.Performing evenly across languages · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- As long as trusted instructions and untrusted retrieved content occupy the same model context, defenses that only change how untrusted text is marked or weighted (instruction hierarchies, spotlighting, content detectors) reduce indirect-injection success but do not remove it; only defenses that isolate contexts or restrict the action set give guarantees, at a cost in the agent's ability to combine information across sources.Following instructions hidden in data · reviewed by AI
- Rewriting a harmful request in an emotional or narrative style (fearful, curious, compassionate) while keeping its intent raises jailbreak success by up to 57 percentage points across 16 instruction-tuned models, contextual rewrites beating fixed templates, and the vulnerability does not shrink with model scale; a preprocessing step that has a second model strip stylistic cues cuts the success rate substantially.Refusing harmful requests under obfuscation · reviewed by AI
- For tool-using agents, a safety harness searched separately for each model-and-domain deployment (jointly evolving a natural-language policy and executable pre/post-tool-call code, screened by a fresh-context adversarial reviewer) reaches a better safety-utility trade-off than fixed expert-designed defenses ported unchanged across deployments, because domain semantics determine which safety relations and trajectory state must be checked while the victim model's failure behavior determines whether a check should be semantic or deterministic and where in execution it must run.Following instructions hidden in data · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model with unsafe rates within 10 points across all ten languages under both attack families.