Large language models can pick the right causal strategy most of the time but fail on the fine print: they identify the high-level design in 79% of cases but correctly specify all design details only one-third of the time, making nuanced research design the bottleneck for automated causal inference.

Main Finding

The authors introduce CausalReasoningBenchmark, a curated, real-world benchmark (173 queries over 132 datasets drawn from 79 peer‑reviewed papers and 3 textbooks) that disentangles identification (research-design specification) from estimation (numeric implementation). By requiring structured identification outputs and providing gold identification JSONs plus reference estimation scripts, the benchmark enables diagnosis of whether failures come from flawed causal reasoning (identification) or from implementation/estimation. A state‑of‑the‑art LLM baseline (gpt-5.3 w/ reasoning) finds the correct high‑level strategy in ≈79% of cases but achieves full identification‑spec correctness in only ≈34%, showing the main bottleneck is nuanced research‑design specification rather than computation.

Key Points

• Scope and sources
  • 173 queries, 132 unique CSV datasets; curated from 79 research papers (political‑science reanalyses) and 53 textbook examples.
  • Covered designs: DiD (67 queries), RDD (44), Conditional exogeneity/selection‑on‑observables (39), IV (22), RCT (1).
• Structured identification schema
  • Agents must output a JSON identification specification listing: strategy label, estimand (ATE/LATE/ATT), treatment(s), outcome(s), controls (minimal sufficient set), and design‑specific fields (e.g., instrument(s) & encouragement flag for IV; running variable & cutoff for RDD; time & group variables for DiD).
  • Benchmarks mark “bad controls” (post‑treatment/mediators/colliders) to detect critical errors.
• Disentangled evaluation
  • Identification scoring: field‑by‑field exact matches (strategy, estimand, variable sets, strategy‑specific fields), plus an overall binary “fully correct” indicator.
  • Estimation scoring: point‑estimate error, relative error, whether predicted estimate lies in gold CI, null‑hypothesis agreement, opposite‑direction flag, CI overlap (Jaccard), SE gap. An auto‑rescaling step can correct unit mismatches before scoring.
• Baseline results (LLM)
  • High‑level strategy correct: ~79%.
  • Outcomes correctly identified: 93.6%; causal quantity correct: 73.4%.
  • Full identification‑spec correctness: ~34% (showing failures in specifying nuanced elements such as minimal adjustment sets or excluding bad controls).
  • Baseline pipeline used the model to inspect CSV, generate and run Python estimation code, and output schema‑conformant JSON; outputs were validated and scored versus gold labels.
• Artifacts and availability
  • Gold identification JSONs and reference estimation code (Python/R) for every query.
  • Public release on Hugging Face (dataset link included in the paper).

Data & Methods

• Data composition
  • Research‑paper subset: 120 queries from 79 papers (large reanalysis corpora for IV, RDD, DiD).
  • Textbook subset: 53 pedagogical examples from three causal‑inference textbooks (adds coverage of conditional exogeneity designs).
  • All datasets supplied as CSV plus metadata files describing columns and study context.
• Task setup
  • Inputs to agents: natural‑language causal question, CSV dataset, metadata.
  • Required outputs: (1) structured identification JSON per schema; (2) numeric estimate and standard error (with the model optionally producing runnable estimation code).
• Evaluation machinery
  • Identification evaluator: exact matches on fields; checks that controls are a superset of the gold minimal sufficient adjustment set and that no bad controls are present; strategy‑specific required fields must be correct.
  • Estimation evaluator: compares predicted effect and SE to gold values, computes absolute/relative errors, CI containment and overlap (Jaccard), hypothesis‑test agreement, and SE gaps. Auto‑rescaling option to address unit mismatches.
• Baseline implementation
  • LLM agent prompted to inspect metadata and CSV, produce the identification JSON, write and run Python estimation code (in a code‑interpreter environment), and return the outputs. Validation and scoring performed automatically.

Implications for AI Economics

• Diagnostic evaluation sharpens priorities for model development
  • The clear separation of identification vs estimation shows that current LLMs largely manage high‑level strategy recognition and numerical execution, but struggle with the fine‑grained, normative parts of research design (e.g., choosing minimal sufficient adjustment sets, excluding post‑treatment controls). For applied economics, this means improving causal‑reasoning modules (formalizing domain constraints, training on structured designs, integrating causal graphs) should be a priority over estimator automation alone.
• Standardization and structured outputs aid automation, replication, and auditability
  • A common JSON schema for identification specifications enables tooling that can automatically verify design validity, run standardized estimators, and produce auditable analysis logs—valuable for reproducible empirical economics and policy evaluation.
• Useful benchmark for agent + econometric library integration
  • The availability of gold estimation scripts and real datasets makes this benchmark a realistic sandbox for agents that combine language models, causal‑inference libraries (DoWhy, econML, statsmodels), and symbolic/graphical causal reasoning. Economists and system builders can use it to test end‑to‑end pipelines while isolating design vs implementation errors.
• Cautions for automated policy analysis
  • The 34% full‑spec correctness (despite 79% strategy recognition) highlights the risk of relying on LLMs to propose research designs or run causal analyses without human oversight. Small mis‑specifications (including bad controls) can produce biased or misleading estimates—critical in policy contexts.
• Research and training opportunities
  • The benchmark suggests concrete directions: (i) training models to produce and validate adjustment sets (possibly via causal graphs), (ii) equipping agents with rule‑based checks for bad controls and post‑treatment exclusion, (iii) methods for uncertainty calibration and unit handling, and (iv) human‑in‑the‑loop workflows where models draft designs and researchers verify them.
• Broader value to economic research
  • By curating real‑world, peer‑reviewed empirical examples (not synthetic problems) the benchmark can accelerate development of trustworthy automation tools for replication studies, education (teaching identification), and scalable reanalysis of applied economics literature—so long as design outputs remain interpretable and auditable.

If you want, I can: - Extract a few illustrative example queries from the benchmark (DiD, RDD, IV) and show the gold identification JSON and typical failure modes observed in the baseline, or - Propose concrete model‑engineering strategies (prompting patterns, verification checks, symbolic/graph modules) to improve identification accuracy for economic applications.