Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?

Language resource level flips sign depending on the prompt: high-resource languages are safer under plain harmful queries but more vulnerable once a jailbreak is applied. English is the safest language with standard queries and the least safe under logic jailbreaking, while Telugu moves from third least safe to safest under both attacks.

The evaluation covers 10 languages (Bengali, Telugu, Swahili; Turkish, Arabic, Korean; German, Chinese, Spanish, English), 6 models (GPT-4o, Sonnet 4, Qwen2.5-7b/14b, Llama3.1-8b/70b), and 2 datasets (HarmBench 200 queries, AdvBench 520), with 2 attacks: logic-expression conversion and the Andriushchenko et al. prompt-and-suffix method.

The prompt-based attack pushes open-source models to near-saturation, e.g. Qwen14b 99.5 in English on HarmBench and 99.42 on AdvBench, while Sonnet4 stays at 00.0 in English but reaches 47.5 in Swahili. Logic jailbreak works better on GPT-4o (61.0 English HarmBench) and poorly on Sonnet4. Within-model unsafe rates differ by more than 50% across languages.

Effects are measured, not argued, with GPT-4o as judge validated against native speakers (Krippendorff's alpha 0.70-0.86). Defenses are language- and model-dependent: self-verification helps stronger models, and a classifier trained only on Qwen7b standard-query responses transfers to jailbreak responses (F1 0.89) but not to other models' standard-query responses (F1 0.68).

Invalid responses were dropped, 10 of thousands of languages covered, 4 model families. Treat English-only safety numbers as uninformative about a model's multilingual attack surface.