FRAME proposes to give leaders measurable, deployment-specific evidence on what AI systems actually do in organizations by pairing large-scale sandboxed traces with contextual metrics; the aim is to reveal where risk and value accumulate in real-world workflows rather than relying on abstract model-centric benchmarks.

Main Finding

FRAME (Forum for Real-World AI Measurement and Evaluation) proposes a scalable, centralized evaluation infrastructure that bridges model‑centric benchmarks and small, local user studies by measuring AI-in-use across realistic workflows. By instrumenting large-scale remote participant panels together with scripted model runs and a metrics translation layer, FRAME treats heterogeneity in human-AI interactions ("user entropy") as a signal rather than noise, producing decision-ready indicators (utility, friction, resilience) that connect model behavior to deployment-level outcomes.

Key Points

• Problem framed: existing evaluation ecosystems (benchmarks, RLHF, red-teaming, isolated pilots) are either scalable but abstract or context-rich but small and fragmented—leaving decision-makers in a “decision‑maker’s dilemma.”
• Core conceptual insight: two interacting sources of variation shape real-world AI outcomes:
  • User entropy (heterogeneity in how people prompt, interpret, adapt, or abandon AI).
  • Model stochasticity (inherent randomness of generative outputs). Modeling both is necessary to understand distributions of real use and higher‑order effects.
• FRAME introduces three evidence layers:
  • Layer 1 — Abstract knowledge: capability benchmarks and lab tests.
  • Layer 2 — Contextual knowledge: user research, pilots, audits (case‑like).
  • Layer 3 — Systematic knowledge: longitudinal, multi‑site measurement (FRAME’s target), analogous to epidemiology/post-market surveillance.
• Two complementary technical components:
  • Testing Sandbox: realistic, scalable evaluation environment combining remote participant panels (task-based scenarios, self-annotation of interactions) with scripted in-silico chatbot runs using the same scenarios.
  • Metrics Hub: translation layer that converts sandbox traces into comparable, deployment-focused indicators (e.g., utility, friction, resilience), aligned to stakeholder decision needs.
• FRAME emphasizes reproducibility, human‑subjects protections (proxy tasks, privacy), and alignment with sponsor needs so organizations can evaluate commercial systems without exposing proprietary data.
• Analogies to prior tech measurement: smartphone telemetry and aggregated GPS studies show how large-scale, context-rich measurement enabled policy and product changes; FRAME aims to play a similar role for AI.

Data & Methods

• Testing Sandbox design:
  • Remote Participant Panels: thousands of panelists complete structured, scenario-based tasks that mimic realistic workflows. Panelists self-report and annotate how they phrased requests, whether and how they adapted outputs, when they switched tools, and any workarounds or abandonment. Individual identities are not exposed; proxy scenarios are used to reduce harm.
  • Scripted Chatbot Runs: automated, in-silico executions of the same scenarios to capture models’ “ideal” or controlled performance.
  • Shared scenarios, detailed logging, and common rubrics link human-in-the-loop and scripted traces for comparison.
• Metrics Hub:
  • Aggregates sandbox traces into indicators that are meaningful for deployment decisions (e.g., time saved vs. rework time, frequency of undetected errors, reliance/over-reliance measures, user friction points, distributional incidence of benefits/risks).
  • Produces regular releases of comparable indicators across sponsors, sectors, and sites.
• Methodological aims:
  • Treat user entropy as measurable signal—capture heterogeneity across users, workflows, settings instead of treating it as noise.
  • Support cross-site comparability while preserving contextual detail.
  • Protect participants via proxy tasks and standard human-subjects protocols.
• Limits and practical design choices (discussed by authors):
  • Use of proxy tasks to avoid exposing participants to high-stakes harms.
  • Trade-offs between ecological validity and safety/scale.
  • The need for standardized protocols and rubrics to enable aggregation across contexts.

Implications for AI Economics

• Better measurement of productivity and costs:
  • FRAME-generated indicators (time saved, rework required, error propagation rates) allow economists to move beyond headline accuracy scores and estimate net productivity effects of AI on tasks and firms, including hidden operational costs.
• Heterogeneity and distributional outcomes:
  • User-entropy data reveal which worker types, tasks, or subpopulations capture gains vs. absorb risks—critical for estimating distributional impacts (wage effects, inequality, occupational transitions).
• Improved causal and quasi‑experimental inference:
  • FRAME’s multi-site, repeatable scenarios and panel structure can support randomized rollouts, difference-in-differences, or instrumental-variable designs for estimating causal impacts of AI deployment at scale.
• Dynamics of adoption and returns to capital vs labor:
  • By measuring friction, reliance, and persistence over time, FRAME data can inform models of adoption dynamics, complementarities/substitution between capital (AI systems) and labor, and whether productivity gains translate into wages or rents.
• Task decomposition and skill-biased technological change:
  • Granular traces on which sub-tasks are automated, require verification, or increase cognitive load allow economists to model task-level reallocation and skill upgrading/downgrading effects.
• Market structure, competition, and procurement:
  • Comparable, real-world performance indicators enable better procurement choices, benchmarking across vendors, and assessment of switching costs and vendor lock-in—affecting market competition and firms’ bargaining positions.
• Regulation, compliance, and liability economics:
  • Deployment-level evidence about error propagation, who notices/corrects errors, and liability shifts can inform cost-benefit analyses of regulation, design of incentives, and optimal compliance regimes.
• Macro forecasting and aggregate productivity puzzles:
  • Aggregated, systematic estimates of friction and realized gains can improve macro forecasts of AI-driven productivity growth by accounting for rework, training costs, and adoption lags that benchmarks miss.
• Externalities and public good considerations:
  • FRAME can surface spillovers (e.g., misinformation propagation, skills erosion) that are important for social welfare calculations and for designing policies to internalize externalities.
• Practical recommendations for economists using FRAME data:
  • Key variables to request or analyze: time-on-task and rework time; quality-adjusted output metrics; incidence and magnitude of undetected errors; persistence of reliance over time; heterogeneity by worker skill, firm size, sector; downstream costs (liability, compliance); user-reported subjective metrics (trust, cognitive load).
  • Use FRAME outputs to parameterize structural models of labor demand, to run counterfactual adoption simulations, and to validate top-down productivity estimates.
• Cautions:
  • External validity still depends on representativeness of panelists and realism of proxy tasks—careful weighting and calibration will be needed when scaling estimates to entire labor markets or sectors.
  • Sponsor incentives and data-access constraints may shape which scenarios are prioritized; transparency around sampling and protocols will be important for robust economic inference.

If helpful, I can (a) draft a short list of specific FRAME indicators economists should prioritize for estimating firm-level productivity impacts, or (b) outline econometric designs (RCT, DiD, event study) that best use FRAME’s sandbox data. Which would you like?