A new benchmark reveals that modern LLMs often produce runnable econometric code but frequently miss the target causal estimate: across 100 synthetic causal tasks, model L2b+ pass rates range from 10% to 88%, and a nontrivial share of executing workflows produce wrong coefficients despite appearing fluent.
CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows
CausalVerify introduces an execution-grounded benchmark showing that runnable LLM-written econometric code is a much stronger predictor of recovering benchmark causal estimates than text-level method or direction agreement, yet many executed workflows still return incorrect treatment-effect estimates.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Existing causal-inference benchmarks for LLMs mostly score method descriptions or whether generated code runs, not whether the executed workflow recovers the target causal estimate. CausalVerify studies this verification problem for structured econometric causal-estimation workflows by separating realistic interpretation from verifiable computation. It pairs 259 published economics papers (reconstructed research question, data description, institutional context) with 100 fixed-seed synthetic scenarios that realise CSV datasets for difference-in-differences, event study, instrumental variables, and regression discontinuity designs. Experiment A (real-paper text agreement) scores method-family and direction agreement against four-LLM consensus labels. Experiment B (synthetic execution) runs model-written R code and checks whether the extracted treatment-effect estimate matches a canonical estimator on the same realised dataset; this execution-grounded correctness layer is L2b+, distinct from L2b, which records only whether the code executes. A calibration arm asks whether self-reported confidence separates correct from incorrect workflows. On Experiment B, seven LLMs reach L2b+ pass rates of 10% to 88% at the default 50% tolerance, and 66 of the 426 workflows that execute (15.5%) return a wrong estimate. Execution ranking (L2b) agrees with L2b+ far better than text-direction scoring (L4): Kendall $τ=0.81$ and Spearman $ρ=0.93$, versus Kendall $τ$ between $-0.20$ and $0.10$ for L4. Llama-3.3-70B-Instruct shows the same qualitative gap, and reported confidence does not reliably separate correct from incorrect workflows. The claims are confined to standardized single-shot workflows in these four design families under the evaluated R backend and model panel; the benchmark does not measure general causal-inference ability. Code, data, cached outputs, and a datasheet are released.
Summary
Main Finding
CausalVerify introduces an execution-grounded benchmark for structured econometric causal-estimation workflows and shows that text-level agreement (naming a method or sign) and code executability are poor proxies for execution-grounded numerical correctness. Under the benchmark’s controlled scenarios, runnable model-written R code often still estimates the wrong causal coefficient; execution success correlates strongly with execution-grounded correctness, but text-direction agreement and self-reported confidence do not.
Key Points
- Benchmark structure
- Two complementary regimes:
- Exp A (real papers): 259 published economics papers used to test method-family (L3) and direction (L4) agreement against 4-LLM consensus labels (text-level diagnostics).
- Exp B (synthetic execution): 100 fixed-seed synthetic DGP scenarios (30 DID, 24 event study, 24 IV, 22 RDD) with realised CSVs and canonical estimators; models generate R code which is executed and scored.
- Calibration arm: models shown their own frozen Exp B outputs are asked to self-report confidence in method/specification/estimate.
- Scoring layers
- L1: output exists.
- L2a: parseable R code present.
- L2b: code executes in the benchmark R environment.
- L2b+: execution-grounded correctness—model-extracted treatment-effect estimate compared to canonical estimator on the same realised dataset; pass if relative error ≤ τ (default τ = 0.5).
- L3: method-family agreement (Exp A).
- L4: effect-direction agreement (Exp A).
- Main quantitative results (selected)
- Seven LLMs evaluated; L2b+ pass rates ranged from 10% to 88% at default 50% tolerance (9%–84% at 25% tolerance reported in appendix).
- 426 workflows executed (L2b); 66 of those (15.5%) returned an estimate that failed L2b+ (i.e., wrong estimate vs canonical).
- Execution (L2b) ranking tracks L2b+ correctness closely: Kendall τ = 0.81, Spearman ρ = 0.93.
- Text-direction agreement (L4) correlates poorly or negatively with L2b+ ranking (Kendall τ between −0.20 and 0.10).
- Self-reported confidence did not reliably separate correct from incorrect workflows.
- Controls and validation
- Canonical estimators and realised datasets fixed before evaluation to avoid penalizing finite-sample deviations to the ideal DGP parameter.
- Coefficient extraction uses a judge instrument (sees executed code, stdout, scenario context but not model identity or canonical estimate); a 50-cell blinded human audit found ~90.9% numeric agreement and ~88.6% agreement on induced L2b+ pass/fail decisions.
- Leakage controls, prompt audits, and multi-stage pipeline reduce grader–respondent circularity.
- Scope & limitations
- Focused on four quasi-experimental families (DID, event study, IV, RDD), single-shot standardized workflows, and an R execution backend.
- Does not cover formulation/data collection, matching/propensity-score/doubly-robust methods, heterogeneous effects, interactive human–model workflows, or other software ecosystems.
- L2b+ is agreement with a benchmark-fixed canonical implementation, not an absolute measure of econometric truth; alternative defensible estimators may be penalized.
Data & Methods
- Data sources
- Exp A: 259 published economics papers (reconstructed research question, data description, institutional context).
- Exp B: 100 synthetic DGP scenarios with fixed seeds producing realised CSVs; canonical estimators precomputed per scenario.
- Models and prompts
- Seven LLMs run on all tasks (same prompts per model); a 4-LLM pool (Claude Opus 4.7, GPT-4o, Kimi, Gemini 2.5 Flash) generated consensus labels for Exp A (with some human audits).
- Execution & scoring pipeline
- Benchmarked models produce R code; code executed via Rscript in a fixed evaluation environment (L2b).
- A coefficient-extraction judge locates the reported treatment-effect estimate in executed output; numerical comparison to canonical estimate yields L2b+ pass/fail.
- Default L2b+ tolerance is relative error ≤ 50%; unscoreable scenarios handled separately (e.g., canonical estimate near zero).
- Validation & robustness
- Regex extraction initially produced scorer errors; final pipeline uses the judge plus human audits.
- Robustness run with Llama-3.3-70B-Instruct reproduced qualitative gap between text and execution-grounded correctness.
- Resources
- Code, datasets, cached outputs and a datasheet released at https://github.com/causalverify/causalverify.
Implications for AI Economics
- For applied econometrics workflows, fluent text and runnable code are insufficient safety checks:
- Deploying LLMs as research-assistants or automation tools requires execution-grounded verification (compare model estimates to independently computed references) rather than relying only on method naming or successful execution.
- Execution is a stronger proxy for correctness than text agreement:
- Systems should prioritize executing model-written analyses and checking numeric results against canonical or agreed-upon implementations as part of validation and review pipelines.
- Self-reported confidence is not a reliable alarm:
- Verbalized/confidence outputs from LLMs should not be trusted in lieu of external verification; rely on quantitative checks or calibrated uncertainty mechanisms.
- Benchmark design lessons for researchers and tool builders:
- Use fixed-seed realised datasets and canonical estimators to make numerical correctness auditable.
- Integrate robust coefficient-extraction and scorer-audit instruments to avoid brittle post-processing failures.
- Consider adding tolerance sweeps and alternative accepted estimators when multiple defensible approaches exist.
- For peer review, reproducibility, and governance:
- CausalVerify-style execution-grounded checks can help flag subtle but consequential errors where code runs but computes the wrong causal quantity—especially important as LLMs are used for drafting empirical analyses.
- Directions for future work
- Broaden coverage: more design families, interactive/iterative workflows, heterogeneous-effects estimands, and other languages/backends (Python/stata).
- Evaluate multi-step human–model collaboration and model-driven specification search (robustness checks, alternative estimators like Callaway–Sant’Anna / Sun–Abraham).
- Improve self-calibration: token-probability-based or task-specific uncertainty quantification that better separates correct from incorrect workflows.
- Study defenses: automated specification testing, cross-checks across independent estimators, and ensemble model comparisons.
Overall, CausalVerify provides a practical and necessary reminder for AI-assisted empirical work: require execution-grounded verification of numerical outputs (not just readable prose or passing code) before accepting causal claims produced with LLM assistance.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The benchmark combines extensive controlled synthetic experiments (100 realised datasets across four design families) and a sizeable real-paper text corpus (259 papers) with careful execution and auditing, yielding reproducible, execution-grounded measurements; however, scope is deliberately narrow (four quasi-experimental families, single-shot workflows, one software backend), canonical estimators may penalize valid alternative implementations, consensus labels for the real-paper arm introduce some circularity, and the scorer/judge is not infallible.
Methods Rigorhigh — The authors use multi-stage separation of construction, response, and grading; fixed-seed DGPs and frozen realised CSVs; canonical estimators precomputed before evaluation; an execution environment that runs model-generated R code; a dedicated coefficient-extraction judge with human audit; leakage controls and robustness checks across multiple LLMs — all of which are strong methodological controls — while acknowledging limitations (R-only backend, choice of canonical estimators, consensus-label circularity).
SampleExp A: 259 published economics papers reconstructed into research question, data description, and institutional context; seven LLMs run on each paper to produce method-family and effect-direction labels (1813 outputs). Exp B: 100 fixed-seed synthetic scenarios producing realised CSVs (30 DID, 24 Event Study, 24 IV, 22 RDD), precomputed canonical estimators on each realised dataset; seven LLMs generate R code which is executed in a controlled Rscript environment; coefficient extraction judged and audited. A calibration arm elicits model self-reported confidence on 646 records.
Themeshuman_ai_collab productivity
IdentificationUses fixed-seed synthetic data-generating processes (DGPs) for four quasi-experimental families (difference-in-differences / event study, instrumental variables, regression discontinuity) and a pre-specified canonical estimator per scenario (e.g. TWFE for DID/event-study, two-stage least squares for IV, local-linear RDD) computed on the realised CSV; model-written R code is executed and the extracted treatment-effect estimate is compared to the canonical realised-data estimate within a relative-error tolerance.
GeneralizabilityExp B uses synthetic DGPs and canonical estimators, so results may not generalize to messy real-world data or alternative defensible estimators (e.g. CS/SA implementations in DID)., Only four quasi-experimental design families are covered (DID, ES, IV, RDD); other causal methods (matching, propensity scores, doubly-robust estimators, mediation, heterogeneous-effects workflows) are out of scope., Evaluation is restricted to single-shot, non-interactive workflows and a specific R execution backend; human-in-the-loop, iterative debugging, or other languages/environments may yield different outcomes., Model panel and consensus-label construction create potential circularity and may not represent all LLMs or future model capabilities., Binary L2b+ tolerance thresholds (default 50%) are arbitrary and conclusions can depend on chosen cutoffs.
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| CAUSALVERIFY evaluates structured causal-estimation workflows using 259 published economics papers for text-agreement tasks and 100 fixed-seed synthetic scenarios for executable coefficient-recovery tasks. Other | positive | Benchmark coverage and evaluation performance across causal-inference workflows |
Reading fidelity
high
Study strength
high
|
n=359
|
| The synthetic execution benchmark covers 30 difference-in-differences tasks, 24 event-study tasks, 24 instrumental-variables tasks, and 22 regression-discontinuity tasks. Other | mixed | Distribution of benchmark causal-design tasks |
Reading fidelity
high
Study strength
high
|
n=100
30 DID, 24 Event Study, 24 IV, and 22 RDD tasks
|
| Across the seven evaluated LLMs, final execution-grounded correctness pass rates ranged from 10% to 88% at the default 50% relative-error tolerance. Output Quality | mixed | Execution-grounded recovery of the canonical treatment-effect estimate |
Reading fidelity
high
Study strength
medium
|
n=7
10% to 88%
|
| At the stricter 25% relative-error tolerance, the L2b+ pass-rate range was 9% to 84%. Output Quality | mixed | Execution-grounded coefficient-recovery pass rate under a stricter tolerance |
Reading fidelity
high
Study strength
medium
|
n=7
9% to 84%
|
| Among the 426 workflows that executed successfully, 66 returned an incorrect treatment-effect estimate, corresponding to 15.5% of executable workflows. Error Rate | negative | Rate of executable workflows producing an incorrect coefficient |
Reading fidelity
high
Study strength
high
|
n=426
66 of 426 workflows (15.5%)
|
| Execution success ranking was strongly aligned with execution-grounded correctness ranking, with Kendall's tau of 0.81 and Spearman's rho of 0.93. Output Quality | positive | Association between code execution success and coefficient correctness |
Reading fidelity
high
Study strength
medium
|
n=7
Kendall τ = 0.81; Spearman ρ = 0.93
|
| Text-direction agreement was weakly or negatively aligned with execution-grounded correctness ranking, with Kendall's tau ranging from -0.20 to 0.10. Output Quality | negative | Association between textual effect-direction agreement and executable coefficient correctness |
Reading fidelity
high
Study strength
medium
|
n=7
Kendall τ = -0.20 to 0.10
|
| Self-reported retrospective confidence did not reliably separate correct from incorrect workflows. Ai Safety And Ethics | null_result | Confidence-correctness discrimination |
Reading fidelity
high
Study strength
medium
|
n=646
|
| Agreement with real-paper consensus labels was not a verified measure of causal correctness; the benchmark uses real papers only for text-level method-family and direction agreement. Other | mixed | Validity and scope of text-based causal-method and direction labels |
Reading fidelity
high
Study strength
high
|
n=259
|
| The L2b+ metric measures agreement with a benchmark-fixed canonical estimator on the realized dataset rather than general causal-econometric correctness. Other | mixed | Agreement with a fixed numerical reference implementation |
Reading fidelity
high
Study strength
high
|
n=100
relative error ≤ 50%
|
| The benchmark is limited to standardized single-shot workflows in four design families and does not measure general causal-inference ability. Other | mixed | Scope and external validity of the benchmark |
Reading fidelity
high
Study strength
high
|
n=100
|