A new double-permutation test fixes inflated DiD significance in small samples where cluster-robust t-tests over-reject; the nonparametric method keeps size correct across error distributions at modest power cost and is available in the sigDD R package.

Main Finding

The paper proposes a dual-margin permutation (doubly randomised) test for inference on the canonical two-group, two-period Difference-in-Differences (DiD) estimator. By independently permuting both the treatment (AFFECTED) and time (TIME) indicators to generate an empirical null distribution of the DiD estimator, the test is finite-sample exact under the sharp null and the maintained exchangeability conditions. In simulation and applications, the doubly randomised test maintains accurate empirical size in small samples where standard parametric procedures (HC0, CRVE1) are often severely anti-conservative; power losses relative to Hansen’s jackknife are bounded and vanish as sample size grows. The method is implemented in the sigDD R package (github.com/profsms/sigDD).

Key Points

• Core idea
  • Generate the null distribution of the DiD estimator by permuting both binary margins (TIME and AFFECTED) independently (Bernoulli(1/2) reshuffling), rather than only permuting treatment assignment.
• Theoretical guarantees
  • Theorem 1: Exact finite-sample Type I error control under the sharp null and exchangeability (i.e., p-value is valid for any n).
  • Proposition 1: Combinatorial gain — dual randomization multiplies the permutation space by a factor of (n choose nT) relative to single-margin fixed-count permutations (or by 2^n under unconstrained Bernoulli reshuffling), producing a denser approximation to the null distribution.
  • Lemma 1: Faster convergence / finer discretization of the null distribution under dual randomization due to the larger permutation space.
  • Entropy justification: independent Bernoulli reshuffling maximises permutation-space entropy and avoids edge-case draws in fixed-margin resampling.
• Practical algorithm (implemented)
• Compute observed DiD estimate (OLS interaction or 2×2 cell-means).
• For B iterations (recommended B = 4,999): draw independent Bernoulli(1/2) labels for TIME and AFFECTED (for each observation), compute permuted DiD estimate, store.
• Form empirical null from permuted estimates; p-value = (1 + number |δ_perm| ≥ |δ_obs|)/(B + 1) (Phipson & Smyth correction).
• Reject if p ≤ α.
• Simulation evidence
  • Monte Carlo with 5,000 replications per configuration across n ∈ {20,…,400} and δ ∈ {0.25,…,2} (sd units).
  • Doubly randomised test preserves nominal size at all sample sizes tested; HC0 and CRVE1 show substantial over-rejection at small n.
  • The main driver of CRVE anti-conservatism is regressor-matrix leverage/effective degrees of freedom rather than error heteroskedasticity—HC corrections do not address this leverage problem.
  • Power: some loss relative to Hansen’s jackknife, but limited and negligible as n increases.
• Empirical validation
  • Applied to four canonical DiD empirical datasets; p-values compared across methods and implementation available in sigDD.
• Positioning and recommended use
  • The method targets regimes where parametric asymptotics are unreliable: small samples, few treated clusters, high-leverage designs, and unbalanced treatment proportions.
  • It complements—not replaces—CRVE/jackknife in large-G settings where those asymptotic methods perform well.

Data & Methods

• Setup
  • Canonical DiD: Yit = α + β TIMEt + γ AFFECTEDi + δ (TIMEt × AFFECTEDi) + εit; test H0: δ = 0.
• Randomization device
  • Bernoulli(1/2) reshuffling of both binary margins per observation (independently for TIME and AFFECTED) used for Monte Carlo permutation draws; fixed-margin permutations are discussed for combinatorial comparisons.
• Inference target
  • Two-sided test using empirical α/2 and 1 − α/2 quantiles of permuted DiD estimates; p-value computed with the standard +1/(B+1) correction.
• Comparators in simulations
  • HC0 (heteroskedasticity-robust t-test), CRVE1 (cluster-robust variance estimator), Bell–McCaffrey refinement, Hansen (2025) jackknife delete-one-cluster estimator with effective degrees-of-freedom adjustment, and single-margin permutation baseline.
• Simulations and robustness checks
  • Controlled Monte Carlo experiments with heavy tails and heteroskedastic errors; 5,000 replications per configuration; permutation B = 4,999 used for approximating null.
• Software
  • sigDD R package implements the doubly randomised test; code and examples on GitHub: profsms/sigDD.

Implications for AI Economics

• Typical AI-economics settings where this is relevant
  • Evaluations of AI policies, pilot programs, firm-level AI adoption, regulatory interventions, or experiments with few treated clusters (e.g., a small number of firms, regions, or industries adopting a new AI policy).
  • Studies with unbalanced designs, high-leverage covariates, or small numbers of treated units—situations common in applied AI policy research.
• Why dual-margin permutation helps in AI-economics applications
  • Finite-sample exactness under the sharp null protects against anti-conservative inference when standard errors (HC or CRVE) are unreliable due to leverage or few clusters.
  • Distribution-free: robust to heavy-tailed or skewed outcome distributions that appear in firm-level or productivity outcomes related to AI adoption.
  • Handles irregular/unbalanced designs naturally; Bernoulli reshuffling avoids problematic bootstrap draws that lack treated clusters.
• Practical recommendations for AI-economics researchers
  • Use dual-margin permutation (sigDD) as a robustness/primary inference method when:
    • Number of treated clusters or distinct groups is small;
    • Treatment assignment is unbalanced or leverages in the design matrix are high;
    • Outcome distributions are non-normal or heavy-tailed.
  • Complement permutation p-values with Hansen’s jackknife and CRVE in larger samples; interpret differences as diagnostic of leverage/effective-dof issues.
  • Remember the scope: permutation inference is exact under the sharp null and exchangeability; it does not solve identification failures (e.g., violations of parallel trends) or heterogeneous-treatment-effect interpretations.
  • Use B large enough (e.g., 4,999) to stabilise p-values; report the permutation algorithm details (Bernoulli vs fixed-margin) and the Phipson–Smyth correction.
• Caveats for AI-economics inference
  • The strict exactness claim relies on the sharp null (all unit-level effects = 0) and exchangeability under the maintained randomization mechanism; in observational DiD, TIME is not literally randomized—permuting TIME is justified only as a randomization-inference device under H0, not as a claim about the data-generating process.
  • When treatment effects are heterogeneous across units, Fisher-type permutation tests may be conservative for average-null hypotheses; interpret results accordingly.
  • Power trade-offs exist relative to variance-estimator-based methods (jackknife), so weigh the costs when sample sizes are moderate-to-large.
• Overall takeaway for AI economists
  • Dual-margin permutation provides a simple, assumption-lean, and computationally feasible tool for credible inference in small-sample DiD settings common in AI-evaluation work. It should be part of the standard robustness toolkit, used alongside parametric and jackknife methods and sensitivity analyses for identification assumptions (parallel trends, heterogeneity).

If you’d like, I can: - Produce example R code showing how to run sigDD on a toy DiD dataset. - Summarise the simulation results (size/power plots or numeric summaries) in more detail. - Draft a short checklist for deciding when to prefer dual-margin permutation in applied AI-economics studies.