A new auditable benchmark finds foundation-model trading agents are highly scenario-dependent and frequently outperformed by a fixed equal-weight portfolio; process telemetry uncovers silent internal failures that terminal returns can mask.
FORESIGHT-9: Prospective and Process-Aware Evaluation of Adaptive Trading Agents
FORESIGHT-9 is an auditable prospective benchmark of nine counterfactual market worldlines that shows adaptive foundation-model trading agents' performance and internal process coherence vary widely by scenario and backbone, with a simple equal-weight baseline outperforming most agent runs.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Retrospective backtests provide a limited test of adaptive trading agents: they cannot rule out historical contamination, expose sensitivity to a single realized market path, or reveal internal degeneration during long-horizon adaptation. We introduce FORESIGHT-9, a prospective and process-aware benchmark built from nine auditable counterfactual stress worldlines branching from a common July 2026 information boundary. Each worldline specifies staged macro-financial events and joint multi-asset anchors; a deterministic generator realizes the trajectories, while observations are disclosed according to in-world time. A common contract standardizes observations and execution while preserving each agent's native adaptation loop. We evaluate two adaptive trading-agent frameworks with two foundation-model backbones across 36 long-horizon runs. Agent rankings vary substantially across worldlines and backbones, and a fixed equal-weight policy outperforms 31 of 36 runs. Process telemetry exposes failures that terminal returns conceal: in one high-return run, the live factor library collapsed while executed holdings converged to the equal-weight fallback, even though decision records continued to report an active factor ensemble. FORESIGHT-9 therefore evaluates not only portfolio outcomes, but whether adaptive agent state and execution remain coherent across alternative futures. We release the worldlines, trajectories, audit traces, and regeneration scripts.
Summary
Main Finding
FORESIGHT-9 is a prospective, process-aware benchmark that evaluates adaptive trading agents across nine auditable counterfactual worldlines (forked from a common 2026-07-15 information boundary). When applied to two agent frameworks (AlphaCrafter and FactorMiner) under two foundation-model backbones (gpt-5.6-terra, deepseek-v4-flash) over 36 long-horizon runs, results show large sensitivity to worldline and backbone choice; a simple equal-weight baseline outperformed 31 of 36 adaptive runs. Crucially, process telemetry reveals internal degenerations (e.g., collapsed factor libraries, silent convergence to fallback execution) that terminal returns can hide.
Key Points
- Benchmark design
- Nine documented counterfactual "worldlines" (WL1–WL9) each with 4–5 staged macro-financial regimes (examples: geopolitical conflict, inflation re-ignition, financial-system fragility, energy collapse, pandemic/automation).
- Shared information boundary at t0 = 2026-07-15; agents see only rows with date ≤ in‑world date (strict leakage control).
- Deterministic generator: stage endpoint anchoring + seeded Brownian-bridge local noise; trajectories and seeds are published for reproducibility and audit.
- Process-aware evaluation
- Records persistent research state: factor lifecycle events (proposed, admitted, evicted), per-decision proposals, executed holdings, and failure reasons.
- Execution separated from decision: a deterministic execution layer applies emitted target weights verbatim; any divergence is auditable.
- Reflection and memory writes are explicit, time-gated, and logged.
- Agents and contract
- Two frameworks evaluated:
- FactorMiner (FM): LLM-driven mining + deterministic admission/combination, capacity-30 factor library, deterministic admission gates (|IC| ≥ 0.007, |ICIR| ≥ 0.084), top-10 IC-weighted combination; fallback = equal-weight 1/15.
- AlphaCrafter (AC): multi-agent miners → screener → trader; similar deterministic gates and auditing but different search modality (proposal-based).
- Common evaluation contract: 10-trading-day decision cadence, long-only fully invested portfolios, 3 bps one-way transaction cost on migrated notional, deterministic execution, guaranteed window advance (calendar never stalls).
- Outcomes and process findings
- Large cross-worldline and cross-backbone variability in cumulative returns.
- Mean cumulative returns (per Table 2): FM–terra +77.8%, AC–terra +40.6%, AC–DS +49.4%, FM–DS +29.7%; equal-weight baseline (EW-15) mean +71.2%.
- Despite adaptive agents producing positive returns, the equal-weight policy outperformed 31/36 agent runs—showing absolute returns alone do not prove valuable adaptation.
- Process telemetry uncovered failures invisible to terminal metrics: e.g., a run with high terminal NAV where the live factor library had collapsed and executed holdings converged to the equal-weight fallback while decision logs still reported active ensembles.
- Reproducibility and release
- Authors release full manifests: worldline panels and narratives, deterministic trajectories, audit traces (factor logs, trade records), warmup and online factor inventories, and regeneration scripts.
Data & Methods
- Data panel
- Warmup window: 2020-01-01 to 2026-07-15 (shared across all runs).
- Worldline horizon: 2026-07-15 → 2035-12-31 (≈2,468 trading days).
- 20-series panel: 15 tradable exposures (equity indices, commodities, crypto, sovereign yields) + 5 observation-only signals (FX indices, VIX). Returns are local-currency percent changes (no FX conversion).
- Worldline generator
- Each stage specified by narrative endpoints (joint multi-asset anchors) and terminal dates.
- Within-stage paths realized by a seeded Brownian bridge anchored at stage endpoints; news publication is time-gated after price rows (price-leads-news rule).
- Worldlines are deterministic given manifest and seed and are auditable against declared endpoints.
- Agent setup & protocol
- Two frameworks × two LLM backbones → 36 runs; each run starts from identical initial capital and identical byte-identical worldline inputs.
- Decision cadence: every 10 trading days; proposals persisted even if gated out; deterministic rebalance rule using forecasted returns and migration cost threshold (rebalance only if e > 3τ, where τ is migration fraction).
- Admission gates (FM and AC admission logic): staged screening by IC/ICIR thresholds and conflict-count routing (k ≥ 2 rejects, k = 1 triggers quality duel, k = 0 admits). Library capacity = 30; combination uses top-10 IC-weighted ensemble preserving sign(IC).
- Execution layer strictly applies target weights; traders cannot place raw orders. Per-decision telemetry logged (observations, tool calls, candidate factors, validation outcomes, proposals, execution results, skip reasons).
- Metrics reported
- Cumulative return, annualized volatility, Sharpe (rf=0), max drawdown, turnover, transaction costs; medians and per-WL decompositions provided.
Implications for AI Economics
- Retrospective backtests are insufficient for evaluating adaptive AI trading agents
- Single realized-path evaluation cannot distinguish overfit memorization from transferable decision policies, nor reveal regime sensitivity or internal degeneration.
- Need for prospective, multi-scenario evaluation in deployment and regulation
- Stress testing adaptive agents should use documented counterfactual futures with time-gated disclosure and deterministic regeneration to audit behavior under alternative macro regimes.
- Regulators and firms should require auditable process telemetry (research state, factor lifecycle, decision vs execution parity) as part of model risk management.
- Simple baselines remain strong and necessary
- Equal-weight/naïve baselines can outperform complex adaptive agents across many plausible futures; adaptive agents must demonstrate consistent, robust improvement over such baselines, not merely positive absolute returns.
- Model/backbone choice matters and is non-robust
- Agent performance ordering can flip across LLM backbones and worldlines — model selection should account for regime sensitivity, not only historical backtest performance.
- Design prescriptions for trustworthy adaptive systems
- Preserve and log persistent research state; enforce deterministic execution with auditable divergence reporting; use conservative admission/eviction rules and explicit fallbacks; require time-gated memory writes and reproducible scenario manifests.
- Research directions
- Quantify relation between process-level failures and long-horizon performance degradation; develop metrics that combine outcome and process coherence for more informative leaderboards.
- Extend prospective benchmarks to sampling designs that reflect decision-makers’ uncertainty weighting (while keeping scenario provenance explicit, not probabilistic priors).
Short actionable takeaway: evaluate adaptive trading agents prospectively across multiple documented counterfactual futures and require process-level audit trails; otherwise, retrospective success can hide brittle behavior that simple baselines already avoid.
Assessment
Paper Typedescriptive
Evidence Strengthlow — Findings are derived from documented synthetic counterfactual worldlines and simulation runs rather than real-world causal variation; the benchmark robustly measures agent behavior under designed scenarios but does not establish causal effects in live markets.
Methods Rigormedium — Design is careful and reproducible (shared information boundary, deterministic seeds, audit trails, standardized contract, multiple worldlines, two agent frameworks and two model backbones) which supports internal validity of the benchmark; limitations include reliance on hand-specified scenario anchors (not sampled from a calibrated probability model), a limited asset universe and long-only constraint, and potential sensitivity to scenario design.
SampleSynthetic evaluation across nine documented counterfactual worldlines branching from a common information boundary (t0 = 2026-07-15) to 2035-12-31; panel of 20 time series (15 tradable exposures: global equity indices, commodities, crypto, sovereign yields; plus 5 observation-only signals); deterministic generator using stage endpoint anchors plus Brownian-bridge bridge noise; two agent frameworks (FactorMiner, AlphaCrafter) each run under two foundation-model backbones (gpt-5.6-terra, deepseek-v4-flash) yielding 2×2×9 = 36 completed runs from identical initial capital and a 10-trading-day decision cadence; published trajectories, factor-lifecycle audit logs, and regeneration scripts.
Themesinnovation adoption
GeneralizabilityWorldlines are designed stress scenarios, not draws from an empirically estimated probability distribution, so results describe behavior under documented scenario spread rather than expected real-world outcomes., Limited asset universe (15 tradable exposures) and long-only, fully-invested constraint restrict applicability to other asset classes or trading constraints., Synthetic market dynamics (anchored endpoints with Brownian-bridge noise) omit real-world market microstructure, order-book dynamics, liquidity shocks, and counterparty effects., Two selected foundation-model backbones and two agent frameworks may not represent the broader space of possible architectures or trading implementations., No real-money live-market validation; execution costs are stylized (fixed 3 bps rule) and may understate slippage or market impact in large-scale deployments.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| FORESIGHT-9 evaluates adaptive trading agents across nine auditable counterfactual worldlines that branch from a common July 15, 2026 information boundary and extend to December 31, 2035. Decision Quality | positive | Coverage and robustness of adaptive-agent evaluation across alternative market futures |
Reading fidelity
high
Study strength
medium
|
n=9
|
| The benchmark evaluates two adaptive trading-agent frameworks under two foundation-model backbones across 36 completed long-horizon runs. Decision Quality | positive | Adaptive trading-agent performance across benchmark conditions |
Reading fidelity
high
Study strength
high
|
n=36
36 runs
|
| Agent performance rankings vary substantially across worldlines and foundation-model backbones. Decision Quality | mixed | Cumulative portfolio return and relative agent ranking |
Reading fidelity
high
Study strength
medium
|
n=36
FM-terra mean return +77.8% versus AC-terra +40.6%; AC-DS +49.4% versus FM-DS +29.7%
|
| The fixed 15-asset equal-weight policy outperformed 31 of the 36 adaptive-agent runs. Decision Quality | negative | Cumulative portfolio return relative to the equal-weight baseline |
Reading fidelity
high
Study strength
medium
|
n=36
31 of 36 runs
|
| FM with the terra backbone achieved the highest mean cumulative return among the four agent-backbone combinations, at +77.8%, while FM with DS achieved the lowest, at +29.7%. Decision Quality | mixed | Mean cumulative portfolio return across worldlines |
Reading fidelity
high
Study strength
medium
|
n=9
FM-terra +77.8%; FM-DS +29.7%
|
| Performance can change sharply for the same framework across backbones and worldlines; specifically, FM-terra returned +138.7% on WL5, whereas FM-DS returned +8.4% on WL5. Decision Quality | mixed | Worldline-specific cumulative portfolio return |
Reading fidelity
high
Study strength
medium
|
n=2
FM-terra +138.7% versus FM-DS +8.4%
|
| Process telemetry identified a high-return run in which the live factor library collapsed and executed holdings converged to the equal-weight fallback, despite decision records continuing to report an active factor ensemble. Ai Safety And Ethics | negative | Coherence between adaptive-agent research state, reported decisions, and executed portfolio holdings |
Reading fidelity
high
Study strength
low
|
n=1
|
| The benchmark's process records can distinguish portfolio outcome quality from internal degeneration or execution inconsistency. Ai Safety And Ethics | positive | Ability to detect and audit process-level failures in long-running adaptive agents |
Reading fidelity
high
Study strength
medium
|
n=36
|
| Across worldlines, WL4 was the most difficult scenario for both agents and the equal-weight baseline, while WL6 and WL8 produced the highest returns for the equal-weight baseline. Decision Quality | negative | Worldline-level cumulative portfolio return and scenario difficulty |
Reading fidelity
high
Study strength
medium
|
n=9
|