A new benchmark shows LLM trading agents fail to adapt: eight of 13 models use fixed strategies in adversarial crypto simulations, while longer ‘thinking’ helps knowledge retrieval but yields no meaningful trading gains.

Main Finding

TraderBench introduces a performance-grounded benchmark for financial AI agents that combines expert-verified static tasks with adversarial, simulation-based trading evaluation. When applied to 12 candidate models (plus two ablation variants), the benchmark reveals three core problems: (1) many agents appear "robust" only because they adopt fixed or inert strategies, not because they adapt to adversarial market manipulation; (2) models show a large conceptual-vs-computational gap in options (good at strategy identification but poor at precise Greeks and risk computation); and (3) LLM-based rubric scoring is substantially more variable across judges than performance-based metrics, making evaluation reliability itself a safety concern.

Key Points

• Benchmark structure
  • Four equally weighted sections (25% each): Knowledge Retrieval, Analytical Reasoning, Options Trading, Crypto Trading.
  • Two novel tracks: adversarial crypto trading (4 progressive market transforms) and decomposed options scoring (P&L, Greeks, strategy, risk).
• Architecture & tooling
  • Two-agent A2A setup: Evaluator Agent + Candidate Agent using six MCP servers (SEC EDGAR, Yahoo Finance, Sandbox execution, Options pricing, Trading simulator, Risk metrics).
  • Performance metrics for crypto: weighted combination of return (35%), Sharpe (30%), win rate (20%), max drawdown penalty (15%).
• Main empirical findings
  • Top model on overall score: Gemini-3-Pro (64.3); leaderboard shows a ~20-point gap between best and worst models.
  • Knowledge Retrieval is the primary driver of cross-model differences (KR range ≈ 9.3–61.6), while Analytical Reasoning and Options Trading show much lower inter-model variance.
  • Tool use & planning matter: enabling "thinking" (chain-of-thought) improved KR dramatically in one ablation (+26.1), web-search MCP added modest KR gains (+4.6).
  • Options gap: consistently high scores on conceptual dimensions (P&L, strategy) vs. very low scores on computational dimensions (Greeks accuracy); mean conceptual-vs-computational gap ~54 points.
  • Crypto adversarial robustness: 7 of 12 models score ~32–34 across all transforms with <1-point variation — indicating fixed/non-adaptive strategies (often inert); 5 models actively trade with higher scores (≈45–52) but some show large performance collapse under noisy/adversarial transforms.
  • Judge reliability: re-scoring identical outputs with three different judge LLMs yields substantial variance (Overall range 11.3; KR range 28.8), while performance-based crypto scoring is judge-invariant (range ≈0.3).
• Limitations noted by authors
  • Current experiments use ~50 tasks sampled from a larger pool and single-run evaluations (no bootstrap CIs reported).
  • The benchmark is extensible, with capability to refresh market data to avoid contamination.

Data & Methods

• Evaluation configuration
  • ~50 tasks per evaluation (42 scored across four sections), stratified sampling (seed 42).
  • Main rubric judge: GPT-5.2 (temperature 0.0). Judge-variance study uses additional judges.
  • 6-hour timeout per model run.
• Candidate infrastructure
  • All candidate models share identical Candidate Agent infrastructure and access to the six MCP servers; this isolates model capability from infrastructure differences.
• Models evaluated
  • 12 candidate models spanning proprietary frontier systems, open-weight, and open-source (e.g., Gemini-3-Pro, Grok 4.1 Fast, GPT-5.2, GPT-4o, several Qwen/Gemma/OSS variants). Two ablations: GPT-5.2+web-search, Qwen3-32B with extended thinking.
• Crypto adversarial transforms (applied to historical price data)
  • Baseline: clean data.
  • Noisy: Gaussian price noise (σ = 2%) + volume spikes.
  • Meta: combined noise with trend mods, false breakouts.
  • Adversarial: coordinated false signals (targeting common technical strategies).
  • Transform weighting in final crypto score: baseline 40%, noisy 30%, adversarial 20%, meta 10%.
• Options scoring
  • Four equal sub-dimensions (25% each): P&L accuracy, Greeks accuracy (tolerance window), strategy quality, risk management.
• Scoring normalization
  • All outcomes normalized to [0,100]; section scores are mean of normalized task scores; overall score is weighted sum (equal weights across sections).

Implications for AI Economics

• Evaluation design matters for deployment and policy
  • Performance-based, simulator-grounded metrics (e.g., realized P&L, Sharpe, drawdown) yield far more reliable and judge-invariant assessments than rubric-based LLM scoring. Economic researchers and regulators should prefer outcome-driven evaluation where possible for financial AI.
• Tool-access & retrieval drive real-world competence
  • The dominant axis of model stratification is tool-enabled Knowledge Retrieval, not raw reasoning. For applied econometric or trading tasks, the interface between models and accurate, up-to-date data sources is a critical bottleneck—improvements in API design, tool parameterization, and model planning will yield outsized gains.
• The "competence mirage" introduces economic risk
  • Models can recommend plausible derivative strategies while miscomputing key quantitative parameters (Greeks). That gap creates a realistic failure mode where models produce convincing but quantitatively unsafe advice—dangerous for automated portfolio construction, hedging, and risk management.
• Adversarial fragility and “inaction as robustness”
  • Many models achieve apparent robustness by being inert or using fixed strategies (e.g., buy-and-hold), which masks inability to adapt to manipulated signals. Deploying such agents could produce fragile market behaviors and systemic risk if the deployment environment differs from the benign baseline.
• Evaluation reliability affects research conclusions and regulation
  • Large judge variance for rubric-graded sections implies that single-judge evaluations can mislead ordering of models and policy decisions. Economic studies using LLMs should report multi-judge calibration, prefer objective outcomes when available, and quantify judge uncertainty.
• Research and policy recommendations
  • Adopt performance-grounded benchmarks (simulations with adversarial scenarios) for financial AI research and certification.
  • Prioritize work on robust tool interfaces: correct parameterization of external pricing/risk calls, improved parsing of high-precision numerical outputs, and planning/control for tool usage.
  • Require multi-judge and human-expert calibration for rubric-based scoring; report variability and confidence intervals.
  • Expand adversarial testing (manipulated signals, microstructure noise) when assessing market-facing agents to avoid false confidence from inert policies.
  • For regulators: condition approval or certification on stress-tested performance across adversarial scenarios and on interpretability/verification of quantitative calculations (e.g., independent Greeks checks).

In short: TraderBench shows that current LLM-based trading agents may look competent on paper (or in static Q&A) but hide critical weaknesses—poor tool use, fragile adaptation, and numerical inaccuracies—that matter for economic outcomes. Robust, performance-based evaluation and improved model–tool interfaces are essential next steps for safe research and deployment in financial markets.