tool-using-llm-trading-agents-frozen-backbone-periodically-rewriting-system CLM-0647For tool-using LLM trading agents with a frozen backbone, periodically rewriting the system prompt from the agent's own decision traces and realized portfolio returns changes tool-use behavior reliably — code-interpreter calls rise from about 1 to 2.6-4.5 per day and regime-relevant metrics (VaR, EMA, RSI, signal normalization) that the fixed-prompt agent never computes start appearing — but the resulting risk-adjusted performance beats the fixed-policy agent in only about half the evaluated model-by-regime settings (best Sharpe and cumulative return among LLM methods in 3 of 6).
Capability: Improving its own scaffolding from failures
Observed on
Daily-close simulated trading over 15 US blue-chip stocks plus cash, one-month windows after each model's knowledge cutoff, GPT-5-mini and Gemini-2.5-Flash, policy updated every 5 .
Sources
- Validated by: Sharpe ratio, cumulative return, max drawdown, win rate and volatility against rule-based baselines (SPY, B&H, MACD, KDJ&RSI, ZMR, SMA), a static tool-calling agent, a tool-free evolving agent, and a strategy-only evolving agent (ATLAS-style), across six market regimes and a 50-day horizon; plus per-day maps of which analytic metrics appear in code-interpreter calls.. Measured on six one-month regimes plus one 50-day window, two backbones, three runs each averaged, with rule-based and three ablation baselines. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM agents on dependent multi-step tasks, success decays geometrically with the number of steps under a single per-step reliability parameter that rises with model scale but stays below 1, so long horizons eventually collapse even for the strongest deployed systems — and the decay tracks step count, not context length, since bounding the context window steepens the decay rather than easing it.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM agents on long-horizon web penetration testing, holding the backbone model fixed and moving global state out of the context window onto a persistent graph of verified facts and fact-conditioned intent edges raises task success most on the hardest tasks, where success depends on facts obtained dozens of steps earlier (88.2% overall and 75.0% hard, vs 44.1% and 25.0% for the strongest shared-plain-text-memory baseline).Completing tasks that take many steps or many hours · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- For LLM agents acting through stateful tools over many turns, unsafe policy-violating completions accumulate with the interaction budget and are roughly doubled when the adversarial user conditions each next request on the agent's actual prior response, so single-turn or short-prefix safety evaluation understates the failure rate.Using the tools it is given · reviewed by AI
- For tool-integrated LLM agents where the attack works by adding attacker-controlled tools, overwriting the agent's visible toolset with the saved user toolset immediately before planning drives attack success to near zero across direct and indirect injection, memory poisoning and backdoor attacks, but its effect on benign task success is model-dependent — raising it for 7-9B open models and lowering it for GPT-3.5/GPT-4.Following instructions hidden in data · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A run where the self-evolved policy shows the same analytic-metric usage and tool-call distribution as the fixed prompt, or where fixed-policy agents match or beat the evolved agent on Sharpe and cumulative return in a majority of model-by-regime settings under the same seeds and universe.