EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

Optimizing an agent safety harness separately for each model and each domain beats one fixed expert-designed defense on the safety-utility frontier. EvoSafeHarness runs a search loop in which a Designer proposes a natural-language policy plus executable code (pre/post tool-call hooks), a fresh-context Criticizer rejects rules keyed to benchmark artifacts, and a staged cascade scores benign utility and direct/indirect attack success separately.

Evaluation covers DecodingTrust-Agent (5 victims x 3 domains, 30 benign, 35 direct, 35 indirect tasks per cell), Agent-SafetyBench, AgentDojo with zero-shot transfer to AgentDyn, and PAIR-style adaptive attacks from AgentCanary.

On DTAP, average ASR falls from 45.6 % to 10.0 % at a 3.3-point utility cost, best score in 14 of 15 cells; Progent reaches 10.5 % ASR but drops utility to 56.4 % versus 79.8 %. On AgentDojo it hits 82.8 % utility at 0.0 % ASR, twice CaMeL's utility at zero ASR, and transfers unchanged to AgentDyn at 75.0 % / 0.0 % where CaMeL completes nothing. Under adaptive attack, ASR moves from 9.7 % at budget 0 to a mean 19.5 % at budget 16.

Evidence is measured with held-out splits invisible to the search, paired McNemar tests rejecting no-change in 13 of 15 cells, and ablations showing removal of the Criticizer costs up to 31 held-out points. Baselines are frozen per cell while the method is searched per cell, so search cost is not matched. Three risk types (client-targeted scams 60.0 %, options solicitation 30.0 %, telecom finance fraud 16.7 %) show no reduction, since harm lies in fabricated content, not observable action relations.

If you deploy harnesses, treat them as per-deployment artifacts rather than portable guardrails.