5 cumulative citations
View corpus contextA psychometric scale derived from model performance lets researchers predict human task completion time from AI benchmark results. Using this mapping, the authors forecast frontier models' human-time capability and find the 50% solvable-task horizon roughly doubles every six months.
BRIDGE: Predicting Human Task Completion Time From Model Performance
BRIDGE fits a two-parameter IRT model to model responses and shows latent task difficulty maps linearly to log human completion time, enabling inference of human task length from model performance and forecasting that the 50% solvable-task horizon expands roughly twofold every six months.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task difficulty. Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across benchmarks. In this work, we propose BRIDGE, a unified psychometric framework that learns a latent difficulty scale from model responses and anchors it to human task completion time. Using a two-parameter logistic Item Response Theory model, we jointly estimate latent task difficulty and model capability from model performance data across multiple benchmarks. We demonstrate that latent task difficulty varies linearly with the logarithm of human completion time, allowing human task completion time to be inferred for new benchmarks from model performance alone. Leveraging this alignment, we forecast frontier model capabilities in terms of human task length and independently reproduce METR's exponential scaling results, with the 50% solvable task horizon doubling approximately every 6 months.
Summary
Main Finding
BRIDGE shows that a psychometric latent difficulty scale inferred from model performance (via a 2-parameter logistic IRT model) aligns tightly with human task completion time: latent difficulty b is approximately linear in log human time (R² ≈ 0.81). After calibrating this mapping on METR human annotations, BRIDGE can predict human task completion times for new benchmarks from model responses alone, and use model-release histories to forecast human-interpretable capability horizons (the 50%‑solvable task length), which it estimates doubles roughly every 6 months — consistent with METR’s exponential-scaling result.
Key Points
- Method: Fit a 2PL IRT model (item discrimination ai, item difficulty bi, model ability θj) to binary model success/failure data aggregated across many benchmarks.
- Calibration: Regress log(human task time) on IRT difficulty bi using METR tasks (those with human time annotations). Result: strong log-linear fit (R² = 0.81); each +1 in b ≈ 2.26× increase in human completion time.
- Prediction: Applying the calibrated mapping to IRT difficulties yields human time estimates for tasks on out-of-distribution benchmarks (SWE-bench Verified, MLE-bench, GDPval, Cybench) without new human studies.
- Forecasting: Within rolling release windows, take the best model’s θ; because 50% success corresponds to b = θ, map θ → human time to obtain a human-task-length horizon. BRIDGE finds exponential growth with a doubling time ≈ 6 months for the 50% horizon.
- Baselines compared: logit(success-rate) heuristic and direct LLM-based time predictors (Gemini 3 Pro, GPT-5.2). BRIDGE aligns better with annotated human time buckets on held-out benchmarks.
- Practicalities: Uses binary success (≥50% success in repeated runs when available), MCMC estimation with hierarchical priors, tolerates sparsity in the model×task matrix (≈51% observed entries in experiments).
- Code: McGill-NLP/BRIDGE (public repository).
Data & Methods
- Core model:
- 2PL IRT: P(success) = σ(ai(θj − bi)). ai captures discrimination; bi is latent difficulty on same scale as θ.
- Parameters estimated by MCMC with hierarchical priors (implementation following Lalor & Rodriguez 2023 / Natesan et al., 2016).
- Calibration:
- METR dataset (170 tasks across SWAA, HCAST, RE-bench) provides human completion-time annotations. Fit log(h) = slope × b + intercept.
- Observed result: slope implies ≈2.26× human time per +1 b.
- Evaluation benchmarks (out-of-distribution validation):
- SWE-bench Verified (software engineering / GitHub issues; coarse human-time buckets).
- MLE-bench (end-to-end ML workflows; 75 Kaggle competitions, three binary success indicators each).
- GDPval (44 occupations; used LLM-as-judge pipeline where needed).
- Cybench (cybersecurity CTF tasks; uses exact-match flag verification).
- Baselines:
- Logit success-rate: transform empirical task and model success rates with logit and calibrate same log-linear mapping to human time.
- LLM estimators: prompt frontier LLMs (Gemini 3 Pro, GPT-5.2) to predict human completion times from task descriptions.
- Metrics and outcomes:
- Strong R² for METR calibration (0.81).
- BRIDGE better matched annotated time buckets on SWE-bench Verified vs heuristics and LLM-prompted estimates (paper reports higher accuracy/F1/kappa for BRIDGE).
- Forecasting reproduces exponential frontier growth (doubling time ≈6 months).
Implications for AI Economics
- Translating model performance into human time creates a practical, interpretable unit for economic impact analyses:
- Enables mapping model capabilities to labor tasks measured in minutes/hours, facilitating estimates of hours-of-work displaced/augmented and productivity gains.
- Allows economists to forecast when classes of human tasks (by time-to-complete) become automatable at target success probabilities, informing timing of labor-market transitions.
- Scalable measurement: BRIDGE reduces the need for repeated, costly human time-annotation studies by enabling transfer of human-time calibration across benchmarks from model-only performance data — this lowers the data-collection barrier for broad economic assessments.
- Policy and investment relevance:
- Faster, quantitative forecasts of task-length horizons can inform policy timing (retraining programs, social safety nets), regulatory planning, and private-sector investment decisions (which tasks/roles to automate first).
- Combined with task frequency and economic value (e.g., task incidence in occupations, wages, GDP shares), BRIDGE outputs can feed models that estimate GDP effects, sectoral disruption, and labor demand shifts.
- Caveats and considerations for economic use:
- Human time is only one dimension of economic importance: value-per-minute, task frequency, substitutability, complementarities, and regulatory constraints all matter for realized economic impact.
- BRIDGE relies on calibration to METR and on the assumption that the psychometric latent scale transfers across domains; calibration should be periodically rechecked as model behavior and benchmarks evolve.
- Tasks unsolved by all models are excluded because IRT difficulty estimates are unreliable there; this may bias visibility over the hardest tail of tasks until models make initial progress.
- Some economic-relevant benchmarks (open-ended tasks, complex human judgment) may require richer outcomes than binary success; integration with graded success metrics is a potential extension.
- Recommended next steps for AI economists:
- Combine BRIDGE time estimates with task incidence datasets (e.g., O*NET, occupation task maps) and wage/GDP weights to produce sectoral automation forecasts.
- Quantify uncertainty from the IRT fit and calibration, and propagate it through economic impact models.
- Use BRIDGE to prioritize targeted human studies on tasks where calibration uncertainty is largest or where economic value per unit time is highest.
Summary: BRIDGE provides a scalable, empirically validated bridge from model performance to human task completion time via IRT + METR calibration. That bridge enables human-interpretable capability measurement and economic-style forecasts (e.g., when tasks of given human durations become reliably solvable), but economics applications should combine BRIDGE outputs with task incidence, value, and uncertainty modeling to estimate real-world impact.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper demonstrates a consistent empirical relationship across multiple benchmarks and reproduces prior exponential-scaling results, which supports the core claims; however, strength is limited by reliance on IRT assumptions (unidimensionality, local independence), the quality and representativeness of human completion-time anchors, potential dataset overlap between models and benchmarks, and risks from extrapolating the fitted relationship beyond observed ranges.
Methods Rigormedium — The use of a two-parameter IRT model is appropriate and methodologically sound for estimating latent difficulty from binary performance data, and the joint estimation + anchoring is a principled approach; nevertheless, the analysis depends on strong modeling assumptions (IRT structure, logistic functional form, linearity with log time), likely sensitivity to selection of benchmarks/models, and limited robustness checks or causal identification strategies to rule out alternative explanations (e.g., training-data leakage, multidimensional task constructs).
SampleModel performance (accuracy/pass rates) collected across multiple benchmarks and tasks for a range of language models (including frontier models over time); a subset of tasks/benchmarks has measured human task completion times used to anchor the latent scale; forecasts use time-series of frontier model performance aggregated onto the anchored difficulty scale.
Themesproductivity human_ai_collab
IdentificationFit a two-parameter logistic Item Response Theory (IRT) model to model performance data to jointly estimate latent task difficulty and model capability; anchor the latent difficulty scale to observed human task completion time using a linear relationship between latent difficulty and log(human time); use the anchored scale to infer human completion times for new tasks/benchmarks and to forecast frontier model capability over time.
GeneralizabilityAnchoring depends on which benchmarks and human samples were used; different task types or populations may show different time–difficulty relationships., Assumes tasks are unidimensional and comparable under IRT; complex/multifaceted tasks may violate this., Human completion times can be noisy and context-dependent (skill, instructions, incentives), reducing external validity., Potential training-data overlap between models and benchmarks could bias inferred difficulties downward for some tasks., Extrapolating the fitted linear relationship and exponential capability trends into the future or to very different task domains is uncertain.
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across benchmarks. Adoption Rate | negative | cost/noise/scale of human task completion time annotations |
Reading fidelity
high
Study strength
low
|
not reported
|
| We propose BRIDGE, a unified psychometric framework that learns a latent difficulty scale from model responses and anchors it to human task completion time. Other | positive | latent task difficulty scale (anchored to human task completion time) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Using a two-parameter logistic Item Response Theory model, we jointly estimate latent task difficulty and model capability from model performance data across multiple benchmarks. Other | positive | latent task difficulty and model capability estimates |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Latent task difficulty varies linearly with the logarithm of human completion time. Task Completion Time | positive | relationship between latent difficulty and human task completion time |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Human task completion time can be inferred for new benchmarks from model performance alone by leveraging the learned alignment between latent difficulty and human completion time. Task Completion Time | positive | inferred human task completion time |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Leveraging this alignment, we forecast frontier model capabilities in terms of human task length. Task Completion Time | positive | frontier model capability expressed as human task length solvability |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| We independently reproduce METR's exponential scaling results, with the 50% solvable task horizon doubling approximately every 6 months. Task Completion Time | positive | 50% solvable task horizon (time until tasks of a given human-length are solvable by models) |
Reading fidelity
high
Study strength
medium
|
doubling approximately every 6 months
|