0 cumulative citations
View corpus contextLarge language models are unstable as trading agents — their actions flip and vary wildly even under deterministic decoding; reframing models to output executable alpha factors (AlphaForgeBench) yields deterministic, reproducible evaluations aligned with quant research workflows.
AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
The paper documents severe behavioral instability of LLMs when used as sequential trading agents and proposes AlphaForgeBench — a factor-generation benchmark that reframes LLMs as quantitative researchers to produce deterministic, reproducible evaluations of financial reasoning and strategy design.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
The rapid advancement of Large Language Models (LLMs) has led to a surge of financial benchmarks, evolving from static knowledge evaluation toward interactive trading simulations. However, existing frameworks for evaluating real-time trading largely overlook a critical failure mode: the severe behavioral instability of LLMs in sequential decision-making under financial uncertainty. Through extensive experiments, we show that when deployed as trading agents, LLMs exhibit extreme run-to-run variance, generate inconsistent action sequences even under deterministic decoding, and frequently produce irrational action flipping across adjacent time steps. We attribute these behaviors to the stateless autoregressive nature of LLMs, which lack persistent memory of prior actions, together with their sensitivity to continuous-to-discrete action mappings in portfolio allocation tasks. These deficiencies fundamentally undermine the reliability and reproducibility of many existing online and offline trading benchmarks. To address these limitations, we propose AlphaForgeBench, a principled evaluation framework that redefines LLMs as quantitative researchers rather than stochastic trading agents. Instead of producing discrete trading actions, AlphaForgeBench requires models to generate executable alpha factors and compose factor-based trading strategies grounded in financial knowledge. This paradigm decouples reasoning from execution mechanics, enabling deterministic and reproducible evaluation while remaining aligned with real-world quantitative research workflows. Extensive experiments across multiple state-of-the-art LLMs demonstrate that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning, strategy formulation, and alpha discovery. Webpage at https://finbrain-lab-hkustgz.github.io/AlphaForgeBench
Summary
Main Finding
Large Language Models (LLMs) are unreliable when used as direct trading agents: they exhibit extreme run-to-run variance, produce inconsistent action sequences even under deterministic decoding, and flip actions rapidly. AlphaForgeBench reframes evaluation by treating LLMs as quantitative researchers that generate executable alpha factors and strategy code. This decouples reasoning from execution, yields deterministic backtests, removes execution-induced instability, and provides reproducible, discriminative measures of LLM financial reasoning and strategy-generation ability.
Key Points
- Failure modes identified for direct-trading LLMs:
- Large run-to-run variance in returns and risk metrics under identical inputs.
- Inconsistent action sequences even with deterministic decoding (temperature=0).
- Rapid action flipping (buy then immediately sell etc.), resistant to prompt guardrails.
- Root causes:
- LLMs are fundamentally stateless autoregressive models (no inherent memory of prior executed actions).
- Sensitivity to continuous → discrete mapping: small continuous input changes can trigger discrete action flips.
- Direct output acts like classification (label current state) rather than sequential strategy optimization accounting for transaction costs, slippage, and stateful constraints.
- AlphaForgeBench design principles:
- Reposition LLMs to produce explicit, executable factor definitions and strategy code (generate_signal function) rather than point-wise trade labels.
- Separate code-generation (stochastic) from execution (deterministic backtest).
- Provide a factor library and allow model-generated novel factors to be dynamically registered.
- Use a fixed prompt template across models to avoid prompt-engineering confounds.
- Empirical claims:
- Applied across multiple SOTA LLMs, AlphaForgeBench eliminates execution-induced instability and yields reproducible outcomes.
- It provides a rigorous, discriminative benchmark for assessing alpha discovery, financial reasoning, and strategy formulation.
Data & Methods
- Dataset construction (two stages):
- Stage 1: Real-world extraction
- Sources: brokerage reports, quant platforms (WorldQuant, JoinQuant), AI-in-finance literature, open-source repos (Qlib, OpenFE), traditional finance publications.
- Automated extraction agent (gemini-3-flash-preview) produced structured factor-strategy records.
- Collected 3,176 factor-strategy entries; evaluation in this paper focuses on 633 single-asset strategies (to isolate signal-generation).
- Stage 2: LLM-augmented query generation
- Constructed 270 additional queries using a 3 × 3 taxonomy (Levels × Grades):
- Levels: Level 1 Logic Translation (fully specified), Level 2 Logic Completion (skeletons with missing params), Level 3 Goal-Oriented Generation (high-level objectives).
- Grades: Easy / Medium / Hard vary conditions and control-flow complexity.
- Constructed 270 additional queries using a 3 × 3 taxonomy (Levels × Grades):
- Stage 1: Real-world extraction
- Evaluation pipeline:
- Prompt format (identical across models): (i) system instruction + data schema (OHLCV + precomputed indicators), (ii) strategy query, (iii) factor-library reference.
- Models must return a self-contained Python function generate_signal(df) that outputs a trading-signal series per the backtest interface.
- Dynamic registration for novel factors generated by models.
- Backtesting:
- Unified deterministic backtest engine; historical daily data across 7 assets spanning crypto and US equity regimes (2021–2026).
- Metrics computed: returns, drawdown, risk-adjusted measures (Sharpe, ARR, MDD, Calmar/CR, Sortino (SOR), volatility), return curves.
- Procedure: run each model multiple times (k = 5) to capture generation stochasticity; report mean ± standard deviation for metrics so cross-run stability is quantified.
- Stratified reporting: overall model ranking, per-asset decomposition, per-level (Stage 2 taxonomy) decomposition.
- Models evaluated (examples shown): GPT-5.2, Gemini variants, Claude, Grok 4.1, Sonnet, Flash, DeepSeek, etc.
Implications for AI Economics
- Benchmarking & reproducibility:
- Many existing online/offline trading benchmarks that evaluate LLMs as execution agents are unreliable because model stochasticity and statelessness inflate variance and break reproducibility. AlphaForgeBench shows one principled alternative that yields reproducible, auditable evaluation.
- Model design and deployment:
- LLMs intended for market interaction should incorporate explicit state, inertia, and optimization objectives (e.g., transaction costs, slippage, holding periods) or be paired with deterministic execution engines to avoid pathologies.
- Treating LLMs as strategy designers (rather than low-level executors) maps better to real-world quant workflows and reduces risk of destabilizing market behavior if deployed.
- Regulatory & audit considerations:
- Executable, deterministic strategy code increases auditability and interpretability; this is important for compliance, risk controls, and explaining model-driven trading decisions to stakeholders.
- Research directions in AI economics:
- Comparative advantage: AlphaForgeBench lets economists and ML researchers measure models’ relative ability to synthesize tradable signals and economic logic rather than measure noisy execution outcomes.
- Market impact modeling: The instability of direct-execution LLM agents implies potential for amplified market noise; simulations and theory should account for behavioral instability of model-based traders.
- Architecture implications: Encourages development of LLMs or hybrids that natively support stateful decision-making, constrained optimization, or integration with reinforcement learning that accounts for sequential costs.
- Limitations and next steps:
- Current evaluation focuses on single-asset strategies (633 entries); portfolio and multi-asset settings (2,172 portfolio + 371 multi-asset entries exist in corpus) are reserved for future work and are necessary to assess allocation skills and cross-asset interactions.
- Backtest realism remains a factor: deterministic backtests increase reproducibility but depend on how well costs, slippage, and market-impact are modeled.
- AlphaForgeBench reduces execution-induced instability but generation randomness remains; future work could explore controlled generation, program-synthesis verification, and hybrid model+engine co-design.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents systematic experimental evidence that LLMs exhibit high run-to-run variance and action instability in trading simulations, and demonstrates that a factor-generation paradigm removes execution-induced variability in those experiments; however, the findings are limited to the models, decoding settings, simulated markets, and prompt designs tested and do not establish broader causal claims about real-world trading outcomes.
Methods Rigormedium — The authors run extensive experiments across multiple state-of-the-art LLMs, evaluate deterministic decoding, and benchmark alternate evaluation paradigms (action-based vs factor-based), which shows careful design; but the paper (based on the abstract) lacks reported details about statistical significance, robustness across many market regimes, specifics on models/versions, and real-world execution constraints (market impact, latency), so important rigor and external-validity checks appear missing or are not described here.
SampleExperimental evaluations use multiple state-of-the-art LLMs deployed as trading agents in online and offline trading simulations and portfolio-allocation tasks; experiments compare run-to-run variability (including deterministic decoding), action-sequence consistency, and instability metrics, and test the proposed AlphaForgeBench that asks models to output executable alpha factors and factor-based strategies rather than discrete actions. (Paper webpage: https://finbrain-lab-hkustgz.github.io/AlphaForgeBench.)
Themesinnovation adoption
GeneralizabilityResults may depend on the specific LLM architectures, sizes, and model versions tested and may not generalize to other or future models., Simulated trading environments may not capture real-world market microstructure, transaction costs, market impact, latency, and regulatory constraints., Performance and stability likely sensitive to prompt engineering, instruction templates, and any few-shot examples used., Findings on portfolio allocation or specific asset classes/time periods tested may not hold across different financial regimes or instruments., AlphaForgeBench's factor-based approach assumes access to realistic factor construction and backtesting infrastructure, which may differ across firms.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| When deployed as trading agents, LLMs exhibit extreme run-to-run variance. Error Rate | negative | run-to-run variance |
Reading fidelity
high
Study strength
medium
|
not reported
|
| LLMs generate inconsistent action sequences even under deterministic decoding. Error Rate | negative | action sequence consistency |
Reading fidelity
high
Study strength
medium
|
not reported
|
| LLMs frequently produce irrational action flipping across adjacent time steps. Error Rate | negative | frequency of irrational action flipping across adjacent time steps |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The behavioral instability of LLM trading agents is attributable to the stateless autoregressive nature of LLMs (lack of persistent memory of prior actions) and their sensitivity to continuous-to-discrete action mappings in portfolio allocation tasks. Error Rate | negative | cause of behavioral instability (statelessness and mapping sensitivity) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| These deficiencies fundamentally undermine the reliability and reproducibility of many existing online and offline trading benchmarks. Research Productivity | negative | benchmark reliability and reproducibility |
Reading fidelity
high
Study strength
medium
|
not reported
|
| AlphaForgeBench redefines LLMs as quantitative researchers rather than stochastic trading agents by requiring models to generate executable alpha factors and compose factor-based trading strategies grounded in financial knowledge. Research Productivity | positive | model output format and role (alpha-factor generation and factor-based strategy composition) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| AlphaForgeBench decouples reasoning from execution mechanics, enabling deterministic and reproducible evaluation while remaining aligned with real-world quantitative research workflows. Research Productivity | positive | determinism and reproducibility of evaluation |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Extensive experiments across multiple state-of-the-art LLMs demonstrate that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning, strategy formulation, and alpha discovery. Research Productivity | positive | elimination of execution-induced instability; quality of benchmarking for financial reasoning, strategy formulation, alpha discovery |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Financial benchmarks have evolved from static knowledge evaluation toward interactive trading simulations. Adoption Rate | null_result | benchmark evaluation paradigm (static knowledge vs interactive simulations) |
Reading fidelity
high
Study strength
low
|
not reported
|