Microsoft 365 Copilot is seen as reliable and easy to use, with administrative employees reporting highest immediate gains and scientists growing more positive over time; benefits concentrate on clearly structured, text-based knowledge work and suggest learning and routinization effects that call for role-specific training and governance.

Main Finding

In a pilot rollout of Microsoft 365 Copilot at a large non-university research organization, employees generally judged the tool as user-friendly, technically reliable, and useful—with the greatest practical gains for clearly structured, text-based tasks. Administrative staff reported higher usefulness and perceived output quality early on, while scientific staff showed significant positive shifts over time (notably in perceived usefulness, ease of use, productivity and workload reduction), consistent with learning and routinization effects. The study highlights the importance of role-specific implementation, training, and governance for sustainable acceptance of generative AI in knowledge-intensive organizations.

Key Points

• Sample and design

  • Pilot in a large non-university research organization (quota-based selection of license holders).
  • Two repeated cross-sectional survey waves: T01 (Nov–Dec 2024), N = 106 (66 science, 40 admin); T02 (Mar–Apr 2025), N = 90 (51 science, 39 admin).
  • Participants were pseudonymized; overlap across waves was small, so analyses are repeated cross-sections rather than longitudinal individual trajectories.

• Measures

  • Acceptance constructs drawn from TAM: perceived usefulness (PU, 5 items, α = .97/.95), perceived ease of use (2 items), output quality, reliability, voluntariness; all items on a −3 to +3 Likert scale.
  • Nine task-specific usefulness items (analyzed descriptively).

• Key quantitative findings (selected)

  • Scientific staff PU increased from 0.42 → 1.09 (d = 0.42, p = .022). Perceived ease of use for scientific staff increased from 0.74 → 1.31 (d = 0.45, p = .016).
  • Administrative staff had higher initial PU (0.94 at T01) and higher output-quality ratings at T01 (0.88 vs 0.26 for science; group d = 0.48, p = .018). Differences largely narrowed by T02.
  • Perceived technical reliability was positive and stable in both groups (admin ≈ 1.13 → 1.15; science ≈ 0.96 → 0.94).
  • Respondents reported low anticipated harm from incorrect outputs (means ≈ −2.3), and high voluntariness of use (means ≈ 2.6–2.8).
  • Task-level pattern: greatest added value for clearly structured, text-based tasks (e.g., drafting, summarization); less perceived benefit for tasks requiring deep domain expertise, experimental design, or compliance/legal judgement.

• Contextual dynamics

  • Eight dated updates to Copilot occurred during the study period (content and technical improvements), which likely affected perceptions over time.
  • The study is exploratory and descriptive; multiple comparisons for task items were not inferentially tested.

Data & Methods

• Recruitment and sample

  • 550 employees were provisioned with Copilot licenses as part of a technical pilot and invited to participate; final analysis samples: 106 (T01) and 90 (T02).
  • Quota nomination by organizational units aimed to reflect institute affiliation, age, gender, role, and prior AI experience; not a random or representative sample.

• Survey instrument

  • Online standardized questionnaire (LimeSurvey), items mainly adapted from TAM (Davis, 1989) and related work.
  • Constructs: perceived usefulness (5 items), perceived ease of use (2 items), output quality (1), reliability (1), harm from incorrect output (1), voluntariness (1), plus nine task-specific usefulness items.
  • Responses coded −3 (“strongly disagree”) to +3 (“strongly agree”).

• Analysis

  • Descriptive statistics by group and wave; group/time comparisons via Welch t-tests (unequal variances) and Cohen’s d effect sizes.
  • Task-specific usefulness analyzed descriptively to avoid multiple-testing inflation.
  • Missing-data rule: exclude cases with >20% missing; listwise exclusion otherwise. Implausible response patterns removed.
  • Limitations explicitly acknowledged: small and non-random sample, self-reported measures, limited power for population inference, repeated cross-sectional design prevents individual-level change claims, potential selection bias toward AI-interested users.

Implications for AI Economics

• Heterogeneous returns by role and task

  • Immediate productivity and quality gains appear concentrated in administrative, routine, structured text tasks. For organizations, this implies a faster, clearer ROI from deploying generative-AI tools in back-office and standardized-document workflows than across all knowledge-work functions.
  • Scientific/creative knowledge work shows growing perceived benefits over time—suggesting that complementary investments (learning, prompts/usage practices, integration into workflows) matter: adoption generates increasing returns as users routinize AI-assisted processes.

• Human capital and training investments

  • The rise in perceived usefulness and ease of use among scientists points to the importance of role-specific training and knowledge-sharing. Economic models of AI adoption should factor in dynamic complementarities: short-run heterogeneity but potential medium-run convergence as firms/train staff.

• Organizational implementation and governance costs

  • Sensitive legal/compliance contexts (data protection, IP, patents) in research organizations mean governance, auditing, and role-based access controls are economically relevant costs. Effective implementation will require protocols, monitoring, and possibly human-in-the-loop verification—reducing pure automation gains but preserving value through safer deployment.

• Labor-market effects and task reallocation

  • The pattern is more consistent with task reallocation and augmentation than immediate displacement: routine, structured tasks are augmented (efficiency gains), while domain-expert tasks remain reliant on human oversight. This supports nuanced projections of labor demand changes: decreased time on routinized tasks, increased emphasis on supervision, prompt-engineering, quality assurance, and higher-order tasks.

• Pricing, licensing, and ROI considerations

  • Given concentrated benefits, organizations should prioritize licencing and deployment where marginal productivity gains are highest (administration, standardized reporting). Cost–benefit analyses must include training, governance, and change management, not just license fees.

• Policy and research implications

  • Policy to accompany AI diffusion should emphasize upskilling, support for role-differentiated adoption, and funding for governance/verification infrastructure in mission-critical institutions.
  • Research agenda: complement acceptance surveys with objective productivity measures, longitudinal within-subject designs, and task-level decomposition to quantify real output gains, error costs, and net labor-hour effects.

Caveats to keep in mind - Results are exploratory and based on self-reports from a non-random, small pilot sample during a period of rapid product updates; findings should be treated as indicative hypotheses rather than definitive causal estimates. Future work should combine experimental or observational productivity data, representative sampling, and longer follow-up to quantify economic impacts precisely.