0 cumulative citations
View corpus contextGoogle’s conversational AI search reduces publisher referrals and worsens user experience: a preregistered field trial of ~1,100 US Chrome users finds AI Mode cut click-through rate by ~18.8 percentage points and reduced trust and satisfaction, whereas removing AI Overviews raised clicks to publishers by ~8.8 percentage points but did not improve perceived usefulness or trust.
AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence
In a preregistered randomized field experiment on Google Search, routing users to Google’s AI Mode substantially reduced click-throughs to external publishers and lowered trust/satisfaction, while hiding AI Overviews modestly increased publisher click-throughs but did not improve user perceptions.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
The integration of generative AI into web search delivers synthesized answers to user queries, changing how people navigate and assess information, while raising concerns about the downstream impacts on publishers who supply the underlying content. We conduct a preregistered field experiment (N=1,100) on Google Search, the dominant online search platform, to estimate the causal effects of AI Overviews and AI Mode on user behavior, perceptions, and publisher traffic. We show that removing AI Overviews and AI Mode increases click-through rates to publishers, while an AI Mode-only experience reduces click-through rates and erodes user experience and trust in information found on Google. These findings show that integrating generative AI into web search reshapes online attention, with economic consequences for the online publishers that sustain both search platforms and the overall information ecosystem.
Summary
Main Finding
In a preregistered randomized field experiment (N = 1,100 eligible; N = 956 completed post-survey) that manipulated Google Search’s AI features for one week, routing users into a fully conversational AI Mode substantially reduced clicks to third-party publishers and worsened user experience and trust. Conversely, removing AI Overviews (No AI Search) increased referral clicks to publishers. Key numeric effects (intent-to-treat unless noted): AI Mode reduced click-through rate by -18.8 percentage points and reduced daily search sessions by -0.92; it also lowered trust (-0.34 on a 7-point scale) and satisfaction/usefulness/agency (≈ -0.6 to -0.7 standard deviations). No AI exposure increased click-through rate by +8.8 percentage points (LATE).
Key Points
- Experimental design
- Three-arm, preregistered RCT using a browser extension: (1) No AI Search (hide AI Overviews and AI Mode), (2) Current Search (no change), (3) AI Mode Search (redirect all searches to Google’s conversational AI Mode).
- 3-day baseline (Current) then 7-day treatment period. Participants were U.S.-based Google/Chrome primary users recruited via Prolific and a university program.
- Analysis: ITT estimates for AI Mode; LATE estimates used for No AI due to partial compliance.
- Main behavioral outcomes
- Click-through rate (to external sites): AI Mode −18.8 pp (95% CI: −22.2, −15.3); No AI +8.8 pp (95% CI: 2.3, 15.3).
- Domain-specific declines under AI Mode (fraction of users clicking): news −12.5 pp; Reddit −21.2 pp; Wikipedia −9.9 pp. Ad clicks fell dramatically (−42.7 pp) because AI Mode did not surface ads during the experiment.
- Search sessions per day: AI Mode −0.92 sessions/day (95% CI: −1.30, −0.55), contrary to the hypothesis that AI increases engagement.
- Minutes per session: AI Mode +0.43 minutes; No AI −0.59 minutes (LATE).
- Search-engine substitution: AI Mode increased switching to competitors (Bing/DDG/Yahoo) by +11.2 pp.
- User perceptions
- Trust in Google search results decreased under AI Mode by −0.34 on a 7-point scale.
- Perceived usefulness, satisfaction, and agency were materially lower under AI Mode (~ −0.6 to −0.7 SD).
- Perceived personalization/relevance also fell under AI Mode (−0.42 SD), opposite of platform claims.
- Compliance & robustness
- Routing to AI Mode succeeded for 94.7% of searches in that arm.
- Hiding AI Overviews was only partially successful: 51.1% of Overviews hidden overall (median participant had 50% hidden). A Google HTML change during the study reduced hiding reliability; hence LATE reported for No AI.
- Sample skew: younger, highly educated, and left-leaning relative to the general U.S. population; baseline averages ~9.2 searches/day and ~4.0 sessions/day.
- Qualitative feedback
- Among AI Mode users, prevalent negative themes: loss of control/agency, difficulty navigating to specific websites, limited links/source diversity, verbosity—consistent with measured declines in satisfaction and trust.
Data & Methods
- Sample and recruitment: 1,444 enrolled; 1,100 made ≥1 search during the 7-day experiment and were in the analysis sample for behavioral outcomes; 956 completed the post-survey (343 No AI, 304 Current, 309 AI Mode).
- Intervention arms:
- No AI Search: browser extension attempted to hide AI Overviews (top, inline, People Also Ask) and prevented routing to AI Mode.
- Current Search: no modification.
- AI Mode Search: redirected searches to conversational AI Mode.
- Pre-treatment baseline: 3 days under Current Search to measure pre-treatment behavior; then 7 days under assigned condition.
- Outcomes measured:
- Behavioral: click-through rate (CTR), clicks to specific domain categories (news, Reddit, Wikipedia, ads), sessions/day, minutes/session, searches/session, question-form query fraction, and switching to other engines.
- Survey: trust, satisfaction, perceived usefulness, agency, personalization/relevance, and head-to-head comparisons in news queries.
- Statistical approach:
- Pre-registered analyses, ITT for AI Mode; LATE for No AI to account for partial compliance.
- Robustness checks: session-level analyses, alternative domain classifications, and heterogeneity tests (LLM familiarity, baseline search intensity).
- Main limitations called out by authors:
- Partial compliance in hiding AI Overviews (external change to Google implementation).
- Short treatment window (7 days).
- Sample may not represent all Google users or international populations.
- Outcome focus on clicks and user-reports, not direct publisher revenue measures.
Implications for AI Economics
- Attention diversion and publisher revenues
- Strong experimental evidence that conversational AI embedded in search substitutes for referral traffic: substantial CTR declines (−18.8 pp) and domain-specific drops (news, Reddit, Wikipedia). If sustained at scale, these reductions can materially cut publishers’ ad and subscription revenues that rely on search referrals.
- Removing synthesized AI summaries increases referral traffic, implying design choices by platforms directly shift monetizable attention away from (or back to) independent content providers.
- Platform-market power and bargaining leverage
- By synthesizing and presenting answers, search platforms can internalize information consumption, strengthening platform leverage in negotiations with content providers and potentially compressing price/risk-sharing options for publishers (e.g., paying for access, demanding licensing, or imposing carve-outs).
- Observed user dissatisfaction with AI Mode complicates platforms’ strategic calculus: while platforms may benefit from internalizing content, negative user experience and increased switching intent (to competitors) create tradeoffs.
- Welfare and productivity considerations
- The experiment documents mixed welfare signals: AI Mode reduced clicks (potentially efficient “good abandonment” if user needs are met) but also reduced trust and satisfaction and increased time per session—suggesting substitution did not clearly improve user welfare and may have harmed it in aggregate.
- For policymakers, the findings imply that adoption of AI features can shrink the public attention economy and potentially reduce the supply of high-quality informational content if publishers lose revenue incentives.
- Policy and business responses
- Potential responses include revenue-sharing/licensing arrangements between platforms and publishers, regulatory scrutiny of platform design choices that divert attention, and interventions to preserve sourcing/transparency (which prior work suggests increases trust).
- Market responses by publishers (paywalls, direct distribution, SEO optimization for AI summaries) and by competing search providers (positioning as link-first or hybrid experiences) are plausible and economically consequential.
- Research gaps and priorities
- Need for longer-term, revenue-linked studies measuring pageviews → ad/subscription revenue to estimate aggregate economic loss to publishers.
- Sectoral heterogeneity: effects may differ by content type (news vs. Q&A vs. technical tutorials) and content quality; differential impacts could re-shape content production incentives.
- Platform design experiments: how different citation, transparency, and linking designs affect the tradeoff between synthesized answers and publisher referrals.
- Broader market impacts: effects on content creation supply, platform competition, and downstream consumer surplus.
Overall takeaway: embedding generative AI into dominant search interfaces materially reduces third-party referrals and degrades user-perceived trust and satisfaction in this experiment. Those effects imply meaningful economic risk to publishers and create important strategic and policy questions about how search platforms should deploy AI features while sustaining the broader information ecosystem.
Assessment
Paper Typerct
Evidence Strengthhigh — A preregistered field RCT in a naturalistic setting provides strong internal validity for causal claims; large sample, power analysis, balance checks, ITT and LATE estimates, and multiple robustness checks support the findings. Weaknesses that slightly temper strength are partial non-compliance (especially in the No AI condition due to an external HTML change) and a relatively short treatment window.
Methods Rigorhigh — Design is a preregistered randomized field experiment with pre-treatment baseline period, power calculation, covariate balance checks, ITT and LATE analyses to handle non-compliance, and a range of robustness checks and heterogeneity analyses; main methodological drawback is substantial and externally caused non-compliance in the No AI hiding intervention, which the authors transparently measure and address with LATE.
SampleUS-based adult Google Search users who use Google Chrome and identified Google as their primary search engine; recruited via Prolific (n≈1,387) and Northeastern University work-study students (n=57). 1,444 installed the extension; 1,100 made at least one search during the 7-day treatment window (analysis sample for behavioral outcomes); 956 completed the post-survey (analysis sample for attitudinal outcomes). Sample skews younger (75% under 45), highly educated (87% with ≥some college), and left-leaning (58% Democrat). Baseline: mean 9.2 searches/day (sd 14.3), 4.0 sessions/day (sd 4.7), 4.1 clicks/day (sd 7.0).
Themesadoption productivity
IdentificationPreregistered randomized controlled trial: participants (N_enrolled=1,444; N_analytical=1,100 with ≥1 search; N_post-survey=956) were randomly assigned, via a browser extension, to one of three conditions (No AI Search, Current Search, AI Mode Search); causal estimates reported as intent-to-treat (ITT) and local average treatment effects (LATE) to account for non-compliance, with balance checks and tests for differential attrition.
GeneralizabilityUS-only sample limits geographic and cross-cultural external validity, Restricted to Google Chrome users who consented to install an extension — may not represent broader population of search users, Recruitment via Prolific and university work-study skews younger and more educated than general population, Short treatment duration (7 days) — does not capture long-run behavioral adaptation by users or publishers, Intervention depended on page markup and experienced partial compliance due to Google HTML changes; effects may differ with other implementations or later platform iterations, May not generalize to non-search contexts or to other search engines/platforms
Claims (13)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Assignment to AI Mode Search reduced click-through rate to external sites by 18.8 percentage points relative to Current Search. Firm Revenue | negative | Click-through rate to external sites |
Reading fidelity
high
Study strength
high
|
n=1100
-18.8 percentage points
|
| Exposure to No AI Search increased click-through rate to external sites by 8.8 percentage points relative to Current Search. Firm Revenue | positive | Click-through rate to external sites |
Reading fidelity
high
Study strength
medium
|
n=1100
8.8pp
|
| Assignment to AI Mode Search reduced search sessions per day by 0.92 sessions relative to Current Search. Consumer Welfare | negative | Search sessions per day |
Reading fidelity
high
Study strength
high
|
n=1100
-0.92 sessions
|
| Assignment to AI Mode Search reduced the fraction of users clicking through to news sites by 12.5 percentage points. Firm Revenue | negative | Fraction of users clicking through to news sites |
Reading fidelity
high
Study strength
high
|
n=1100
-12.5pp
|
| Assignment to AI Mode Search reduced the fraction of users clicking Reddit by 21.2 percentage points. Firm Revenue | negative | Fraction of users clicking Reddit |
Reading fidelity
high
Study strength
high
|
n=1100
-21.2pp
|
| Assignment to AI Mode Search increased minutes per search session by 0.43 minutes. Task Completion Time | positive | Minutes per search session |
Reading fidelity
high
Study strength
high
|
n=1100
0.43 minutes
|
| Assignment to AI Mode Search increased the fraction of users searching on a competitor engine by 11.2 percentage points. Market Structure | positive | Use of competitor search engines |
Reading fidelity
high
Study strength
high
|
n=1100
11.2pp
|
| Assignment to AI Mode Search decreased trust in information on Google by 0.34 points on a 7-point scale. Consumer Welfare | negative | Trust in information on Google |
Reading fidelity
high
Study strength
high
|
n=956
-0.34 points on a 7-point scale
|
| Assignment to AI Mode Search reduced perceived usefulness of search responses by 0.59 standard deviations. Consumer Welfare | negative | Perceived usefulness of search responses |
Reading fidelity
high
Study strength
high
|
n=956
-0.59 sd
|
| Assignment to AI Mode Search reduced satisfaction with search by 0.73 standard deviations. Consumer Welfare | negative | Satisfaction with search |
Reading fidelity
high
Study strength
high
|
n=956
-0.73 sd
|
| Assignment to AI Mode Search reduced perceived agency over search responses by 0.66 standard deviations. Consumer Welfare | negative | Perceived agency over search responses |
Reading fidelity
high
Study strength
high
|
n=956
-0.66 sd
|
| Assignment to AI Mode Search decreased perceived personalization and relevance of responses by 0.42 standard deviations. Consumer Welfare | negative | Perceived personalization and relevance of search responses |
Reading fidelity
high
Study strength
high
|
n=956
-0.42 sd
|
| The experiment found no evidence that exposure to No AI Search affected trust in information on Google. Consumer Welfare | null_result | Trust in information on Google |
Reading fidelity
high
Study strength
medium
|
n=956
|