Autonomous LLM agents can generate tradable equity signals, but a single backtest misleads: re-running the discovery engine across historical decision dates and scoring admitted factors on productivity, performance and novelty exposes fragility and varying rankings across systems.

Main Finding

The paper defines and formalizes "Agentic Empirical Asset Pricing" (AEAP)—LLM-based agents that autonomously run at least one full scientific-discovery cycle (hypothesis → formalization → evaluation) over asset-pricing data—and establishes methodological foundations and an evaluation standard for such systems. For factor discovery specifically, it proposes a reference architecture, a multi-part validation gate (productivity, performance, novelty), and a novel outer-loop evaluation that re-executes the entire discovery process at historical decision dates (to test whether the discovery process generalizes, not only a single discovered output). It implements SEADS (a concrete AEAP factor-discovery system) and evaluates it against five re-implemented baselines on two US equity panels, finding no single metric consistently ranks systems and arguing AEAP evaluations must score discovery systems (not only their outputs) along multiple axes.

Key Points

• Definition of AEAP:
  • An AEAP system autonomously executes at least one full cycle of hypothesis → formalization → evaluation on asset-pricing data with no human performing those steps. This distinguishes AEAP from ML predictors and LLM copilots that leave human decisions in the loop.
• Core building blocks (B1–B5) for any AEAP system:
  • B1 Hypothesis generation (LLM proposes mechanism/rationale).
  • B2 Formalization into a checkable artifact (code/formula).
  • B3 Execution/estimation on point-in-time data (sandboxed tool use).
  • B4 Evaluation gate (statistical or LLM-judged decision mechanism).
  • B5 Memory and iteration (persistent state across cycles).
• AEAP evaluation principles (P1–P3):
  • P1 Point-in-time data: inner loop must use only features available at each decision date.
  • P2 Point-in-time process: the inner discovery loop must be re-executable at multiple historical decision dates (outer loop).
  • P3 Out-of-sample-only results: headline numbers must come from splits untouched by either loop.
  • The paper emphasizes that most existing systems satisfy P1 but not P2; re-running the discovery loop historically is crucial to distinguish lucky runs from reliable processes.
• Factor-discovery reference architecture:
  • Modular design: generator (Planner/Proposer), sandboxed implementation/execution, multi-stage validation gate, persistent library, and optional combinator for portfolio construction.
  • Schema-agnostic executor design to run across different characteristic panels without pipeline redesign.
• SEADS system (concrete instantiation):
  • Planner produces structured seeds (exploit vs explore); Proposers expand seeds into formula/specs.
  • Sanity checks include deterministic AST-based firewall, sandboxed execution restricted to statically read columns, and an LLM Alignment Checker making a line-by-line rationale→code trace.
  • Validation gate: ordered suite of independent checks (duplicates/self-collapse, coverage, Rank-IC t-stat with Bonferroni correction, out-of-decade regime consistency, novelty against a held-out reference panel, partial-IC vs top-K controls, Fama–MacBeth t against same controls, sign agreement across checks). All thresholds are set ex-ante.
  • Self-evolution: mutate near-passes using failure-stage feedback, re-run through the same gate for bounded rounds.
  • Persistent state: baseline characteristic stats, append-only candidate outcome log, natural-language lessons, and an admitted-factor library. A simple combinator (Ridge, α=1) ranks stocks into a long–short portfolio for system-level performance scoring.
• Evaluation metrics for discovered pools:
  • Productivity: total admissions per fixed candidate budget (300 candidates used in experiments).
  • Performance: mean per-factor out-of-sample Sharpe (and related OOS metrics).
  • Novelty: per-admitted-factor maximum correlation against union of the system’s other admits and a held-out reference set, averaged across the admitted set.
  • The three bars are jointly necessary: performance alone can reward redundancy; productivity alone can reward rare luck; novelty alone can reward noise.
• Empirical experiment and findings:
  • SEADS is evaluated against five re-implemented baselines (agentic and non-agentic) on two US panels (a JKP global-characteristics subset and a CRSP/Compustat primitives panel checked for novelty relative to a 45-characteristic reference).
  • No single metric ranks all six systems consistently, motivating multi-axis evaluation.
  • The paper surfaces negative findings and limitations that point to evaluation pitfalls for AEAP systems (e.g., systems that only backtest outputs risk accepting processes that were lucky or leak information).
• Broader claim:
  • The need to treat an autonomous discovery system as an object of evaluation in its own right (not just its final outputs) generalizes beyond factor discovery to any agentic scientific discovery.

Data & Methods

• Data panels:
  • Two US equity panels:
  • JKP global-characteristics subset (≈400 characteristics used in experiments).
  • CRSP/Compustat primitive-features panel (~87 variables), novelty-checked against a 45-characteristic reference panel (Bryzgalova et al., 2025).
  • Periodicity: monthly (asset-periods (i, t) with features xi,t observed at close of period t, and next-period excess returns r_{i,t}).
• Formal problem framing:
  • Candidates are symbolic scoring functions fθ mapping xi,t → s_{i,t} (often rank-normalized). Rank-IC ρt = Spearman(s·,t, r·,t) is a core test statistic.
  • Discovery seeks an admitted set G that is out-of-sample valid and non-redundant relative to a disclosed reference set Z.
• Reference architecture and implementation:
  • Re-implemented five baseline systems from the recent literature for direct comparison; distilled a shared architecture and implemented SEADS according to it.
  • Key engineering safeguards: AST-based look-ahead firewall, sandboxed execution with column access restrictions, and LLM alignment checks to prevent mis-specified code/rationale mismatches.
• Validation gate and thresholds:
  • Multi-stage, independent checks (structural, core statistical, robustness) applied in order, with admission requiring passing all checks.
  • Rank-IC thresholds use Bonferroni correction; out-of-decade splits used to test cross-regime generalization.
  • Novelty measured against an external, held-out reference set Z (to avoid circularity).
• Rolling-context / outer-loop evaluation:
  • The entire discovery pipeline (Planner → Proposer → Execution → Gate → Admit) is re-executed at multiple historical decision dates while carrying forward persistent state, enabling assessment of the discovery process’ reliability over time.
• Scoring:
  • Systems evaluated jointly on productivity, per-factor OOS performance, and novelty metrics; also scored on portfolio-level results via the combinator where applicable.
  • Admission budget (300 candidates) shared across systems to normalize productivity comparisons.
• Reproducibility and ex-ante design choices:
  • Gate thresholds and pipeline choices fixed before running experiments; pipeline is schema-agnostic so identical code runs across panels.

Implications for AI Economics

• New evaluation target: autonomous discovery systems must be evaluated as processes, not only by backtesting their single reported outputs. The outer-loop (rolling re-execution) is essential to distinguish robust discovery from lucky or leaked admissions.
• Standardization need: defining AEAP and its building blocks enables shared vocabulary and reproducible baselines—critical for comparability, benchmarking, and scientific accumulation in AI-driven economics research.
• Guardrails for agentic discovery:
  • Point-in-time enforcement, deterministic AST-based firewalls, sandboxed execution, and explicit novelty reference sets are practical safeguards against information leakage and adaptive overfitting.
  • Multi-stage, independently motivated validation gates reduce the risk that a single metric overstates discovery quality.
• Design trade-offs:
  • Memory/persistence and self-evolution can improve productivity but raise composition and overfitting risks; they must be designed and evaluated with the same rigor applied to model training.
  • Schema-agnostic executors enhance portability across datasets and strengthen the external validity of AEAP systems.
• Policy and institutional effects:
  • Regulators, asset managers, and research institutions should require AEAP systems to demonstrate process-level reliability (rolling re-execution and point-in-time fidelity) before trusting them to influence capital allocation or public claims.
  • Public benchmarks and held-out reference panels (analogous to the Z sets used here) should be adopted for community-wide evaluation of agentic discovery systems.
• General scientific impact:
  • The paper’s principles and architecture apply beyond finance: any autonomous scientific-discovery agent benefits from the same point-in-time, re-executable-process, and out-of-sample-only evaluation discipline to avoid conflating lucky outputs with reliable discovery processes.
• Open questions & limitations highlighted by the paper:
  • How to set admission gate thresholds optimally without over-constraining creativity.
  • Residual risks tied to backbone-data vintage and other domain-dependent choices.
  • The paper reports negative findings and practical pitfalls that future AEAP designs must address (e.g., insufficient multi-criterion gates, failure to re-run across historical dates), suggesting further work on robust evaluation curricula and benchmarks.

If you’d like, I can: - Extract and summarize the SEADS validation-gate checklist and thresholds in a compact table. - Produce a short checklist practitioners can use to evaluate an AEAP system against P1–P3 and B1–B5.