Agent-SafetyBench: Evaluating the Safety of LLM Agents

Tool-using LLM agents fail on behavioral safety far more than on content safety, and no current model is close to reliable: across 16 agents, none scores above 60% safe.

Agent-SafetyBench puts agents in 349 simulated interactive environments with 2,000 test cases spanning 8 risk categories and 10 annotated failure modes, built partly by refining samples from R-Judge, AgentDojo, ToolEmu, ToolSword, InjecAgent and GuardAgent (876 cases) and partly by GPT-4o augmentation (1,124 cases). Safety labels come from a Qwen-2.5-7B-Instruct scorer finetuned on 4,000 manually labeled interaction records.

Claude-3-Opus leads at 59.8 total, Qwen2.5-7B-Instruct trails at 18.8, average 38.5. The split is stark: average content safety 68.4 versus behavior safety 30.4, with the "Spread unsafe information" category at 15.6. Failure modes M2 (calling tools with missing information, 18.1) and M7 (using tools flagged as unverified, 12.5) are worst. Defense prompts listing the failure modes help stronger models slightly and weak models not at all; Claude-3.5-Sonnet stays below 70%.

Evidence is a measured benchmark with human review, cross-validation, and a scorer validated at 91.5% accuracy on 200 Gemini-1.5-Flash interactions versus 75.5% for GPT-4o. Environments are simulated, not real; no finetuning defense was tested.

Treat agent tool-use safety as unsolved by prompting.