LLM nudges help discovery, but only in narrow circumstances: non-local proposals increase yield when hypothesis spaces are compressed, proposals move orthogonally out of the explored span, and the remaining signal aligns with targets; randomly novel jumps or full‑rank hypothesis spaces yield no hybrid advantage.

Main Finding

When the hypothesis space is spectrally compressed (most candidate variation concentrated in a few directions), structured local search suffers diminishing returns. Combining structured local search with LLM-generated non-local proposals (a "hybrid" system) helps only if three geometric conditions co-occur: (i) significant spectral compression (1 − reff/N large), (ii) non-local proposals escape the explored span by a nontrivial orthogonal distance d⊥, and (iii) the escaping component is aligned with the target signal (high residual signal alignment, RSA). Random novelty (orthogonality without alignment) increases coverage but not predictive yield. Each of the three conditions is individually necessary for hybrid advantage; their interaction explains when LLM-guided exploration is worth the cost.

Key Points

• Spectral compression: define effective rank reff(ΣA) of the archive correlation matrix; when reff ≪ N, many nominal hypotheses are redundant and local search yields shrink roughly in proportion to reff (Proposition 1).
• Local search limitation: structured enumeration recombines existing directions and cannot generate components orthogonal to the current span S; hence its yield is bounded by the effective dimension of S.
• Three geometric quantities govern hybrid gains:
  • Compression severity: 1 − reff/N (spectral quantity).
  • Escape distance d⊥(z, S) = ∥(I − PS) z∥ (metric quantity).
  • Residual Signal Alignment (RSA) = cosine between the orthogonal residual and target direction y (angular quantity).
• Predictive novelty (PredNovelty) is an empirical estimator of RSA (e.g., ICIR of the orthogonal residual).
• Theorem 1: If any of 1 − reff/N → 0, d⊥ → 0, or RSA → 0, the hybrid yield advantage → 0. Therefore all three are individually necessary.
• Principle 1 / multiplicative diagnostic: UsefulExploration(z) ∝ (1 − reff/N) · d⊥(z, S) · RSA(z). Empirically, a multiplicative model captures aggregate variation (high condition-level R2).
• Empirical results:
  • Synthetic controlled spectra: hybrid outperforms structured search under strong compression; advantage disappears near full rank.
  • A-share factor archive (5,647 stocks, real factors): hybrid gains concentrated in weak-eigenvalue directions; hybrid advantage negatively correlated with reff/N (Spearman ρ ≈ −0.72).
  • LLM-SRBench / symbolic regression: similar pattern—hybrid gains where expression space is compressed.
  • Directed vs random tests: retrieval-guided (semantic) seeds with comparable d⊥ outperform shuffled/random seeds; random orthogonal jumps have near-zero yield despite large d⊥.
• Practical stopping rule: monitor reff/N, mean d⊥, and RSA (or PredNovelty); stop directed non-local exploration when any approaches failure thresholds.
• Caveat: the conditions are necessary but not sufficient; there is stochasticity and estimation noise (individual-level R2 modest; condition-level much stronger).

Data & Methods

• Framework: three operators composing hybrid discovery:
  • Tjump: propose non-local seeds (LLM-generated).
  • Tlocal: structured enumeration around seeds (existing base factors and operators).
  • Tverify: filter by predictive strength, redundancy, orthogonal coverage, and complexity (e.g., IC/ICIR thresholds, correlation limits).
• Key formal objects:
  • ΣA: correlation matrix of archive A; eigen-spectrum {λi}; effective rank reff = exp(−∑ pi log pi) with pi = λi / ∑ λj.
  • Residual f⊥ = f − projA(f); RSA(f) = ⟨f⊥, y⟩/(∥f⊥∥ ∥y∥).
  • Discovery yield measured by counts of candidates passing ICIR/acceptance and redundancy limits.
• Datasets and testbeds:
  • Financial factors: daily OHLCV for 5,647 A-share stocks (2010–2026). Factor discovery trained 2010–2018, validated 2019–2021, evaluated 2022–2026. Real factor sweep using ≈3,638 real A-share factors for compression-level analysis.
  • Synthetic experiments: controlled eigenvalue spectra (reff/N ∈ {0.1, 0.3, 0.5, 0.8, 0.95, 0.98}) with planted signals allocated to strong or weak eigen-directions.
  • Symbolic regression: LLM-SRBench (158 equations) and SRBench tasks.
  • Operational sanity check: OpenML public-tabular experiment to test budget-allocation implication.
• Baselines:
  • Pure structured enumeration (no LLM).
  • Pure LLM proposals evaluated directly (no structured expansion).
  • Random perturbation (random orthogonal jumps).
  • Hybrid (LLM seeds → structured expansion → verification).
• Metrics: IC, ICIR, acceptance rate, discovery breadth (accepted factors), and spectral diagnostics (reff/N, mean d⊥, PredNovelty as RSA estimator).
• Key empirical procedures:
  • Signal planting: place predictive signal in either top-eigen directions or weak eigen-directions to test where hybrid helps.
  • Directed vs random: compare retrieval-guided LLM seeds, shuffled seeds, and random orthogonal seeds at similar d⊥ to separate orthogonality vs predictive alignment.

Implications for AI Economics

• When to deploy LLM-guided exploration (practical budget allocation):
  • Use spectral diagnostics to decide whether to allocate budget to non-local LLM proposals. If reff/N is close to 1, or mean d⊥ ≈ 0, or PredNovelty ≈ 0, stop LLM-driven exploration—it is unlikely to increase yield.
  • Heuristic multiplicative score: H(z) = (1 − reff/N) · d⊥(z, S) · PredNovelty(z). Prioritize seeds with higher H for costly non-local search/compute.
• Designing automated discovery systems:
  • Shift LLM prompting/objectives from generic novelty to targeted proposals optimizing expected RSA (semantic guidance toward weak but target-bearing directions).
  • Integrate verification (Tverify) that explicitly measures orthogonal residuals and PredNovelty before committing expensive local expansion.
• Economics of R&D and compute allocation:
  • The framework gives a principled way to trade off compute/agent costs against expected marginal yield of exploration. Under compression, directed non-local proposals have higher marginal returns; under high-rank regimes, marginal returns are near-zero.
  • Prevent wasteful exploration: random novelty (or indiscriminate LLM sampling) is costly and ineffective—organizations should invest in retrieval, grounding, or learned proposal models that increase RSA.
• Market-relevant interpretation:
  • The Factor Zoo and similar archives can saturate: many published hypotheses are nominally distinct but spectrally redundant, explaining diminishing alpha from pure enumeration.
  • Alpha discovery efforts can benefit from spectral monitoring: target weak eigen-directions where structured archives under-represent predictive variation.
• Policy and research implications:
  • For fund managers and research teams, the framework operationalizes when to combine human/structured priors with LLM creativity versus when iterative local refinement suffices.
  • For academic policy, the results caution against equating volume of hypotheses with independent discovery—metrics should account for effective dimension.
• Limitations and open directions:
  • Conditions are necessary not sufficient; RSA and PredNovelty are estimated with noise—operational thresholds need calibration by domain and sample size.
  • Future work: design LLMs or learned proposal mechanisms that explicitly optimize expected RSA, tighter cost–benefit optimization under verification costs, and extensions to non-linear/structured target signals.

Summary takeaway: LLMs can materially improve discovery only when they generate directed, predictive novelty that both escapes the compressed span and aligns with the target; spectral diagnostics (effective rank, escape distance, RSA) provide an actionable decision rule for whether and how to allocate exploration budget.