A new, auditable telemetry instrument can compute firms' AI adoption stages from existing production logs and distinguish common stall modes in synthetic tests. Its thresholds and real-world validity, however, are not yet proven and require deployment-level calibration.

Main Finding

The paper introduces adoption telemetry: a computable, model-explicit method for measuring enterprise AI adoption by mapping production usage signals to staged change-management milestones. It presents NANTE — a concrete five-stage operationalization (Notice, Attempt, Navigate, Transform, Embed) with openly published telemetry thresholds — and an open-source reference implementation that proves computability by distinguishing a healthy adoption cohort from five synthetic adoption-failure modes. The paper is explicit that the proposed thresholds are unvalidated constructs (demonstrating feasibility, not empirical validity) and outlines a research agenda to calibrate them against real outcomes.

Key Points

• Measurement gap: current measurement traditions (agent evaluation/observability, enterprise usage dashboards, product analytics, change management) each miss what practitioners call adoption — the population-level progression from access to changed work behavior.
• Adoption telemetry definition and commitments:
  • Object: measures cohorts’ progression through an organizational change (not agent quality or raw activity).
  • Substrate: uses production telemetry already emitted by deployed systems (invocations, sessions, turns, task outcomes).
  • Interpretation: explicit staged change model with computable predicates and open thresholds.
  • Properties: computable (mechanical rules), model-explicit (open thresholds), population-level (cohort-focused, not individual scoring), intervention-mapped (diagnoses map to intervention classes).
• NANTE (Notice, Attempt, Navigate, Transform, Embed): a five-stage operationalization that operationalizes stage membership as predicates over telemetry (frequency, consistency, task embedding signals). Threshold values are published as falsifiable proposals.
• Reference implementation: open-source code demonstrates the method on synthetic populations with known ground truth, separating a healthy cohort from five characteristic modes of adoption failure.
• Limits emphasized by authors: the work shows computability, not validity. Thresholds are unvalidated and require production deployment data and empirical calibration.
• Scope: most directly applicable to human-initiated assistants (copilots, chat UIs); extensions for human-in-the-loop agents are discussed but need further work.
• Privacy/ethics design: cohort-level measurement (minimum-size floors) to avoid per-person surveillance and reduce compliance/privacy concerns.

Data & Methods

• Literature synthesis: compares four measurement traditions (agent evaluation/observability, enterprise usage dashboards, product analytics, change management) and identifies the structural gap.
• Formal framework: unifies pre-deployment evaluation gates, production telemetry, and change-management staging into a single instrumented system.
• NANTE operationalization:
  • Stages defined as computable signatures over production signals (e.g., invocation counts, session structure, recurrence / consistency, task outcome integration).
  • Thresholds are explicitly published so they can be tested and falsified.
• Reference implementation:
  • Open-source code (no URL in the excerpt) that ingests simulated telemetry and applies NANTE predicates.
  • Synthetic-population experiments: generate cohorts with known “ground-truth” adoption behaviors, including one healthy cohort and five adoption-failure modes; validate that the implementation classifies cohorts correctly (demonstrates discriminative computability).
• Methods borrowed/related: process-mining style use of event logs as authoritative behavioral record; product-analytics methods (activation funnels, cohort retention) adapted to population-level change staging.
• Empirical limitations: experiments use synthetic data; no calibrated results from real enterprise deployments presented in this paper.

Implications for AI Economics

• Better micro-level empirical inputs: Adoption telemetry can provide richer, stage-based behavioral measures (depth and embedding of AI use) beyond simple active-user counts. These measures can improve microeconometric analyses of how AI affects productivity, worker tasks, and firm outcomes by distinguishing shallow sampling from structural workflow change.
• Improved ROI and diffusion estimates: Economists and valuation analysts can use stage-progression signals (e.g., share of teams at Transform/Embed) as leading indicators for sustained productivity gains and durable P&L impact, rather than relying on headline deployment counts or license activations that overstate economic adoption.
• Calibration of production-function effects: Stage-based telemetry can be used to parameterize models that map AI adoption to labor substitution/complementarity, task reallocation, and total factor productivity, improving counterfactuals in policy and firm-level studies.
• Forecasting abandonment and sunk-cost risks: Diagnosing stall types (the paper maps failure modes to interventions) can inform expected abandonment rates, maintenance costs, and renewal risk — useful for investors, procurement decisions, and macro projections of AI diffusion.
• Policy and market-level measurement: If adopted at scale (with appropriate privacy safeguards), NANTE-like, open thresholds could standardize cross-firm measures of “deep AI adoption,” improving comparability across studies and informing regulatory/competition policy that depends on actual changes in firm behavior.
• Research agenda relevant to economists:
  • Validate telemetry thresholds against firm-level outcomes (productivity, revenue, worker hours) using field deployments and linked administrative data.
  • Develop causal identification strategies tying stage progression to economic impacts (e.g., difference-in-differences around rollout waves, instrumental variables for exogenous exposure).
  • Aggregate cohort-level signals to industry- or macro-level measures while accounting for firm heterogeneity and selection into pilot/testing phases.
  • Study strategic responses and gaming risks (firms might optimize for measured thresholds); design robust metrics or complementarities (survey/perception measures) to mitigate gaming.
• Caveats for application:
  • Current thresholds are constructs needing empirical calibration — do not treat NANTE’s threshold values as proven signals of productivity or P&L impact yet.
  • Privacy and governance constraints matter: cohort-level design mitigates some concerns, but cross-firm aggregation and linking to outcomes require careful legal and ethical handling.
  • Representativeness: enterprises that allow instrumentation or data sharing may not be representative, so inference to broader populations requires attention to selection bias.

Suggested next steps for AI economists interested in this work - Collaborate on pilots that link NANTE stage outputs to firm accounting or productivity measures to validate thresholds and estimate stage-specific economic effects. - Explore extensions for human-in-the-loop and agentic systems (as the paper notes) and quantify how stage definitions change when AI initiates actions. - Combine telemetry-derived stages with survey instruments (perceptions, managerial support) to model the complementary roles of observed behavior and reported readiness in generating economic returns.