arxiv-2609-01519 SRC-0675 · paper

When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation

Created: 2026-09-01 · Ingested: 2026-10-24

Interactive simulations increasingly evaluate policies in markets populated by language-model agents. Their outputs can look economic---prices, profits, consumer surplus, and welfare---without instantiating the behavior named in the claim. We audit this risk in a multi-turn buyer--seller testbed for configurable hotel transactions. An initial implementation reported welfare gains from two marketplace guardrails of +87.4, +35.0, and +28.8 across a Qwen2.5 1.5B--14B ladder. It also gave guarded and unguarded agents different offer schemas and choice procedures. Holding the schema and buyer choos

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsIn multi-turn LLM buyer-seller market simulations, a measured welfare gain attributed to a platform guardrail can be mostly an artifact of the scaffold the guardrail was bundled with: holding the offer schema and buyer choice rule identical across guarded and unguarded cells shrank or reversed the effect (+87.4, +35.0, +28.8 became +7.2, -13.9, +23.8 across a Qwen2.5 1.5B-14B ladder), and single-generation effects on the most favorable profiles shrank from a mean +229 to +37.6 with three generations per cell, with generation residuals accounting for 49.9% of variation.