You can get valid, debiased inference from multiple imperfect AI annotations without costly gold labels: by exploiting conditional independence (with rich observed features) and tensor-decomposition identification, the DMM estimator recovers latent labels and yields asymptotically normal estimates for common downstream regressions provided at least three conditionally independent proxies.

Main Finding

The paper introduces DMM (Debiased inference with Multiple imperfect Measurements), a method that enables consistent, asymptotically normal (i.e., inferentially valid) downstream estimation when the researcher only has multiple error-prone AI-generated labels and no gold-standard labels. Identification is achieved by exploiting conditional independence of the multiple proxies given the unobserved true label and rich observed features (e.g., text embeddings). Estimation uses semiparametric, Neyman-orthogonal debiased moments so that flexible machine-learning estimation of nuisance components is allowed while preserving valid inference for a wide class of downstream estimands (linear/logistic/GLM, dependent or independent categorical variables, etc.).

Key Points

• Problem: AI-measured variables (e.g., LLM annotations) are error-prone; naïvely treating them as true induces substantial bias and invalid CIs, even when accuracy is high.
• Core idea: Replace gold-standard labels with three or more imperfect proxies and identify the latent true label by array/CP (Kruskal-type) decomposition under a conditional independence assumption.
• Conditional independence assumption: proxies are independent conditional on the latent true label and observed unit-level features (Z). This weaker/structured assumption permits heterogeneous, unit-varying misclassification while allowing shared error sources only insofar as they are captured by Z (e.g., embeddings, difficulty measures).
• Identification: builds on Dawid–Skene / Kruskal / repeated-measure latent-variable literature; identification is nonparametric given J ≥ 3 proxies and the conditional independence structure.
• Estimation & inference: constructs a semiparametric debiased moment estimator (Neyman-orthogonal) that permits slow ML convergence rates for nuisance components (conditional confusion matrices), yielding consistency and asymptotic normality for β and valid CIs.
• Scope: accommodates more than three proxies (theory on efficiency gains), binary and multicategory labels, and a broad class of downstream analyses expressible via moments (most MLEs/GLMs).
• Diagnostics & practice: proposes strategies to make conditional independence plausible (use diverse proxy sources, randomize prompts, condition on rich embeddings and difficulty metrics) and overidentification tests when J>3 to probe assumption violations.
• Relation to gold-label methods: complementary trade-offs — DMM avoids costly gold labels but requires stronger structure on proxies; design-based supervised learning / prediction-powered inference need gold labels but impose no proxy independence assumptions. Methods can be combined when some gold labels are available.
• Empirics: simulations show valid coverage and efficiency improvements when adding imperfect but reasonably accurate proxies; empirical validation shows robustness when conditional independence holds approximately.

Data & Methods

• Data setup:
  • True (latent) categorical label X* (binary or multiclass).
  • Observed: outcome Y (downstream), covariates W, and J ≥ 3 imperfect proxies X(1),...,X(J) (e.g., different LLM annotations, human annotators).
  • Optional observed unit-level features Z (rich text embeddings, difficulty scores) used for conditioning.
• Identification:
  • Uses CP / three-way array decomposition and results from Kruskal, Hu, Allman–Matias–Rhodes, Dawid–Skene: the joint distribution of the proxies factorizes given X* and Z, so one can recover the conditional confusion matrices and latent label distribution nonparametrically.
  • Key identifying assumption: for each value of Z, proxies are conditionally independent given X* (no residual shared error after conditioning on Z).
• Estimation:
  • Estimate nuisance objects: conditional confusion matrices (classification rates) p(X(j)|X,Z) and the latent label probabilities p(X|Z), using flexible ML methods as needed.
  • Construct a debiased moment estimator (influence-function / orthogonal moment) for the downstream parameter β so that first-stage estimation errors have limited impact (Neyman orthogonality).
  • Asymptotic theory: consistency and asymptotic normality under mild regularity and convergence-rate conditions on the nuisance estimators.
  • Extensions: treatment when annotated variable is outcome or predictor; analytic results on efficiency as J increases; overidentification tests when J>3.
• Diagnostics & implementation advice:
  • Condition on rich covariates Z to capture shared difficulty / systematic sources of error.
  • Use heterogeneous proxies (different LLM families, model instantiations, human + machine) and randomize prompts/temperatures to reduce correlated errors beyond Z.
  • Statistical checks: overidentification tests, residual dependence checks, sensitivity analyses.
• Empirical evaluation:
  • Simulations: DMM achieves nominal coverage and can gain efficiency by adding imperfect but informative proxies.
  • Validation study (paper): DMM performs well when conditional independence approximately holds.

Implications for AI Economics

• Practical option when gold labels are costly: For many economic applications that rely on large-scale text/image annotations (sentiment, topics, policy stance, event coding), DMM offers a scalable alternative to collecting validation labels, provided researchers can (a) obtain 3+ diverse proxies and (b) measure and condition on features that explain shared annotation difficulty (embeddings, length, language, etc.).
• Study design recommendations:
  • Collect multiple, diverse proxies: different LLMs, model families, prompt variants, and/or human annotators to maximize independent signal and enable overidentification checks.
  • Record rich unit-level metadata (Z): embeddings, difficulty scores, length, language, quality metrics; these are crucial to plausibly satisfy conditional independence and for accurate nuisance estimation.
  • If feasible, randomize prompts or other labeling choices to reduce unobserved dependence across proxies.
  • When possible, combine small gold-label samples with DMM (hybrid approach) — gold labels relax the proxy assumptions and can improve robustness/efficiency.
• Inference and replicability:
  • DMM reduces reliance on up-to-date single models (mitigating sensitivity to model deprecation) by pooling multiple annotations, but correctness hinges on conditional-independence plausibility; authors should report diagnostics and sensitivity tests.
  • For policy or regulatory work where stakes are high, DMM is useful but should be accompanied by robustness checks (overidentification tests, sensitivity to Z inclusion) or complemented with some gold labels.
• Limitations and caveats for applied economists:
  • Stronger assumptions than gold-label approaches: DMM requires that all shared error sources be captured by observed Z or be absent — violations induce bias.
  • Requires at least three informative proxies and adequate variation in those proxies; if proxies are near-identical or all share the same unobserved bias, identification fails.
  • Computational and modeling complexity: estimating conditional confusion structures and building orthogonal estimators requires more modeling infrastructure than naïve approaches.
• Recommended usage guidance:
  • Use DMM when gold labels are infeasible/expensive and you can collect multiple diverse proxies plus rich Z.
  • If small gold validation is available, prefer or combine with DSL/PPI; if not, apply DMM but include and report conditional-independence diagnostics and sensitivity analyses.
  • For publication/reproducibility: describe proxy diversity, Z definition, diagnostics, and any overidentification test results so readers can assess plausibility.

Summary judgement: DMM is a practically useful, theoretically grounded tool for economic researchers who need large-scale AI-based measurement but lack validation labels. It trades the cost of labeling for assumptions about the structure of proxy errors — a reasonable trade in many applied settings if researchers intentionally design the annotation process and collect rich auxiliary features to make the conditional independence assumption credible.