AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

Safety training that blocks harmful chat requests does not carry over to tool-using agents: several frontier models will carry out explicitly malicious multi-step agent tasks with no jailbreak at all, and a chat jailbreak template transfers to the agent setting without degrading agent capability.

AgentHarm contains 110 hand-written malicious agent behaviors (440 with augmentations) across 11 harm categories, using 104 synthetic tools implemented in Inspect, with an average of 3.53 functions per behavior, matched benign counterparts, and hand-written grading rubrics that use an LLM judge only for narrow subchecks. Models tested include GPT-3.5-Turbo through GPT-4o, 4 Claude models, 3 Gemini, 2 Mistral, and 3 Llama-3.1 sizes.

Without attack, Mistral Large 2 scores 82.2% harm with 1.1% refusals and GPT-4o mini 62.5% with 22% refusals; Claude 3.5 Sonnet refuses 85.2%. The template attack lifts GPT-4o from 48.4% to 72.7% (refusals 48.9% to 13.6%) and Claude 3.5 Sonnet from 13.5% to 68.7%. Jailbroken non-refusal scores nearly match benign ones (Claude 81.9% vs 82%; GPT-4o 84.2% vs 89.9%). Best-of-5 at temperature 1 reaches 90.8% for Mistral.

Effects are measured, with a benign capability baseline and prompting ablations (CoT, ReAct, refusal prompt), but 1 jailbreak template, 1 attempt by default, English only, no multi-turn attacks, and synthetic tools that make tasks easier than real misuse.

Treat agent refusal as a separate property from chat refusal when evaluating deployments.