multi-turn-llm-buyer-seller-market-simulations-measured-welfare-gain-attributed CLM-0617mechanismsingle paperReviewed by AI
In multi-turn LLM buyer-seller market simulations, a measured welfare gain attributed to a platform guardrail can be mostly an artifact of the scaffold the guardrail was bundled with: holding the offer schema and buyer choice rule identical across guarded and unguarded cells shrank or reversed the effect (+87.4, +35.0, +28.8 became +7.2, -13.9, +23.8 across a Qwen2.5 1.5B-14B ladder), and single-generation effects on the most favorable profiles shrank from a mean +229 to +37.6 with three generations per cell, with generation residuals accounting for 49.9% of variation.
Capability: Whether the measurement made the finding
Observed on
Qwen2.5-Instruct 1.5B/3B/14B at 4-bit, self-play buyer and seller, 30-60 synthetic hotel profiles, two question rounds, temperature 0.2; repeated-generation probe is post-hoc on fo.
Sources
- supportsWhen Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce EvaluationValidated by: Paired profile-level Both-None welfare differences with 20,000-draw profile bootstrap, exact 16-assignment sign-flip placebo (p=0.50), variance decomposition, and scripted profit-maximizing and inefficient-bundling seller positive controls showing opposite welfare signs under identical buyer-surplus gains.. Measured within one model family and one synthetic hotel testbed; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- For tool-using LLM trading agents with a frozen backbone, periodically rewriting the system prompt from the agent's own decision traces and realized portfolio returns changes tool-use behavior reliably — code-interpreter calls rise from about 1 to 2.6-4.5 per day and regime-relevant metrics (VaR, EMA, RSI, signal normalization) that the fixed-prompt agent never computes start appearing — but the resulting risk-adjusted performance beats the fixed-policy agent in only about half the evaluated model-by-regime settings (best Sharpe and cumulative return among LLM methods in 3 of 6).Improving its own scaffolding from failures · reviewed by AI
- For LLM agents on dependent multi-step tasks, success decays geometrically with the number of steps under a single per-step reliability parameter that rises with model scale but stays below 1, so long horizons eventually collapse even for the strongest deployed systems — and the decay tracks step count, not context length, since bounding the context window steepens the decay rather than easing it.Completing tasks that take many steps or many hours · reviewed by AI
- In a deterministic 365-day e-commerce business simulation of 1,000-4,000 agent steps, end-of-year assets do not predict the six underlying operating competences: across 18 frontier and open-weight models the top earner ranked 16th of 18 on fraud avoidance and below another model on profit per tool call, and the best negotiator finished mid-pack on assets.Completing tasks that take many steps or many hours · reviewed by AI
- In a 20-year deterministic football-management simulation with hidden state and roughly 340-400 decision stops, what separates frontier models' final scores is managerial behavior rather than deliberation volume: cutting slow-payoff investment as the horizon ends (Spearman -0.58 with late-minus-mid investment shift), keeping cash deployed rather than idle (-0.50), and opening contract renewals early (+0.45) track the score on every seed, while token spend is uncorrelated under every accounting (r_s -0.19, all p > 0.38).Completing tasks that take many steps or many hours · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication in which guarded and unguarded cells already share schema and chooser, with several generations per profile-condition, reproduces welfare gains of the original magnitude and sign across model sizes.