Locally deployable open-weight LLMs can handle the bulk of cohort-study data cleaning, reaching up to 88% task completion on a 20-task benchmark; this indicates a practical path to AI-assisted research pipelines without sending sensitive data to cloud services.

Main Finding

Open-weight LLMs running locally on consumer-grade hardware can substantially automate realistic longitudinal survey data-preparation tasks. In a benchmark of 20 tasks (102 derived variables) drawn from the Next Steps cohort, contemporary 31–35B-parameter models achieved high coverage and accuracy (best model balanced performance ≈ 85.9%, average task completion up to 87.9%), showing a viable path for AI-assisted, governance-compliant workflows where data cannot be sent to cloud APIs.

Key Points

• Benchmark scope: 20 realistic data-preparation tasks from a longitudinal study (Next Steps), covering demographic, education, health, and socioeconomic variables; ground-truth created by experienced cohort researchers.
• Outputs: agents produced R scripts and executed them locally; evaluation compared resulting datasets cell-by-cell to ground-truth.
• Evaluation metrics:
  • Completeness: % of expected variables produced.
  • Correctness: % of produced variables meeting agreement threshold (categorical ≥95% cell match; continuous normalized RMSE ≤1e-4).
  • Balanced performance = Completeness × Correctness.
  • Task-level: share of fully-completed tasks and average task completion.
• Model/evaluation setup:
  • Agentic framework: SmolAgents (lightweight, no native tool-calling required).
  • Models required ≥100k token context windows; 3 agentic runs per task; temperature = 0.8.
  • Detailed vs “lite” prompts were tested; main results use detailed prompts aligned to human ground-truth decisions.
• Models evaluated (examples & deployment tiers): Gemma 4 (31B, dense), Qwen3.6-35B-A3B (MoE), Qwen3.5-35B (MoE), GPT-OSS-20B (MoE), Devstral-24B (dense), Qwen3.5-9B, Mistral-3-14B, Gemma 4 E4B (failed to produce structured outputs).
• Top quantitative results (detailed-prompt benchmark):
  • Gemma 4 31B: Completeness 99.7%, Correctness 86.2%, Balanced 85.9%, Complete tasks 75.0%, Avg task completion 87.9%.
  • Qwen3.6-35B-A3B: Balanced 81.7%, Complete tasks 73.3%, Avg task completion 87.6%.
  • Mid/Lower: GPT-OSS-20B Balanced 77.8% (avg task completion 81.0%); Mistral-3-14B Balanced 52.6% (avg 62.0%).
  • Even weaker models often produced most variables (high completeness), but correctness separated performance.
• Error patterns and limits:
  • Reliable on concepts with standard definitions (sex, ethnicity, BMI, language).
  • Failed or weak on tasks requiring complex, survey-specific metadata interpretation: wave-specific logic, taxonomy compression, multi-input derived variables; housing tenure was not fully completed by any model.
  • Task success negatively correlated with prompt length, script length, and number of operations (r ≈ -0.6 to -0.75).
• Reproducibility and transparency: framework and code released publicly as RRBench (https://github.com/UCL-ARC/RRBench).

Data & Methods

• Ground-truth dataset: Next Steps Core (cleaned, research-ready longitudinal data for ages ~14–32), scripts written and validated by cohort researchers. Data stored in wide format across multiple sweeps.
• Task design: 20 tasks packaged with input files, JSON metadata (variable names, labels, value labels), and detailed prompts specifying expected derived variables and coding decisions. Total 102 target variables.
• Agent setup:
  • SmolAgents orchestrated iterative code generation, execution, debugging within a closed local environment.
  • LLMs generated R code, which agents executed; failures triggered iterative fixes by the agent.
  • Each task run 3 times per model; temperature 0.8 to avoid agent “sticking” during debugging.
• Models: only open-weight models released within previous 12 months and supporting large context windows were included; deployment tiers categorized by hardware feasibility (lightweight → consumer → high-end consumer).
• Evaluation:
  • Variable-level: completeness, correctness (95% categorical agreement threshold; numeric RMSE threshold ≤1e-4), balanced score.
  • Task-level: % tasks fully completed, average within-task completion.
  • Downstream analyses: tested how discrepancies affect applied analyses (e.g., regressions on BMI and income) for variable sets with high/medium/low LLM performance (details summarized qualitatively in results).
• Sensitivity analyses: varied correctness thresholds, tested lite prompts, and assessed withholding missing-value handling to probe robustness.

Implications for AI Economics

• Productivity and labor substitution
  • Potential to materially reduce repetitive preprocessing labour in quantitative social and health sciences (authors estimate large duplicated effort across studies). High-performing open-weight models could cut time spent on routine cleaning and variable construction, lowering marginal research costs per project.
  • But gains are heterogeneous: standard constructs (BMI, sex) are easier to automate than bespoke, metadata-heavy tasks. Human oversight remains essential for complex or bespoke derivations.
• Cost structure and deployment incentives
  • Local open-weight models allow institutions to avoid cloud-provider costs and governance barriers around sensitive data. Up-front hardware costs (suitable GPUs, memory for large-context models) are balanced against avoided per-token cloud fees and compliance costs.
  • There is a cluster of viable model sizes: ~31–35B models achieve near state-of-the-art accuracy while still running on high-end consumer hardware (e.g., 24–32GB GPUs). This creates a market opportunity for institutional procurement of local inference infrastructure and managed open-weight stacks.
• Quality, risk, and value-of-information
  • High completeness but lower correctness on some models shows a risk of undetected errors propagating into analyses. Economic value of automation depends on error rates acceptable for downstream inference; small miscoding can lead to large analytic biases, implying non-trivial human validation costs remain.
  • Institutions will need to weigh expected labor savings against validation overhead; formal cost-benefit analyses should include the marginal reduction in expert time, costs to acquire/maintain hardware, and the risk-adjusted costs of errors.
• Market and labor implications
  • Demand shifts: greater demand for AI-fluent data custodians and validators rather than pure manual cleaners. Skills premium may shift toward designing prompts, auditing outputs, and integrating agent workflows.
  • Distributional effects: lower-barrier institutes (with fewer resources) might adopt smaller models with lower accuracy, potentially widening quality gaps unless shared infrastructure or consortia are established.
• Governance and competition concerns
  • Local open-weight models mitigate one barrier (data privacy) but raise questions about standardization, benchmarking, and certification of model behavior on sensitive datasets — potential role for institutional benchmarks (like RRBench) and regulatory guidance.
  • For applied economics, the ability to run powerful models locally reduces reliance on a small set of cloud LLM providers, with competitive implications and possible downward pressure on API pricing for specialized services.
• Research priorities for AI economics
  • Quantify time- and cost-savings per project across task types and model classes; estimate ROI for hardware + maintenance vs cloud fees.
  • Model the value of human validation workflows (how much reviewer time per automated variable) to compute net labor savings and residual risk.
  • Study adoption dynamics across institutions with different compliance constraints and budgets — potential for shared TRE-compatible model hosting or cooperative procurement.
  • Measure downstream analytic impact (bias/variance) across many real studies to translate model accuracy into economic value and risk metrics.

Repository - Benchmark and code: https://github.com/UCL-ARC/RRBench