Generative search engines give inconsistent and heavy‑tailed citation visibility: identical queries return varying domains and citation counts follow a power law, so single‑run snapshots overstate precision; repeated sampling and confidence intervals reveal much apparent concentration is sampling noise.

Main Finding

Citation visibility measured from generative search engines is stochastic and heavy-tailed: identical queries produce varying responses and cited domains across runs, and citation counts follow a power-law distribution with substantial sample-to-sample variability. Therefore single-run point estimates of citation share or prevalence are misleading; visibility metrics should be treated as estimators with uncertainty and reported with confidence intervals. Many apparent inter-domain differences vanish once measurement uncertainty is accounted for.

Key Points

• Generative search platforms are non-deterministic: the same query at different times can yield different answers and different cited domains.
• The authors analyzed three platforms (Perplexity Search, OpenAI SearchGPT, Google Gemini) on three consumer-product topics.
• Two sampling regimes were used: daily collections across nine days and high-frequency sampling at 10-minute intervals.
• Citation counts across repeated samples follow a power-law (heavy-tailed) distribution — a few domains are cited often, many are cited rarely.
• Bootstrap-based confidence intervals show wide uncertainty: many domain-level differences that look meaningful in single-run snapshots fall within the measurement noise.
• Rank stability analysis across the whole citation distribution shows instability not only at the tail but across frequently cited domains; rankings shift substantially across samples.
• Practical conclusion: single-run visibility metrics give a falsely precise view of domain performance; uncertainty must be quantified and reported.

Data & Methods

• Platforms: Perplexity Search, OpenAI SearchGPT, Google Gemini.
• Topics: three consumer-product topics (paper focuses on consumer-product verticals; topics not specified here).
• Sampling designs:
  • Multi-day sampling: one collection per day over nine days to capture day-to-day variation and medium-term drift.
  • High-frequency sampling: repeated queries at ten-minute intervals to capture short-term stochasticity.
• Analyses:
  • Empirical distributional analysis showed citation counts follow a power-law form.
  • Bootstrap resampling was used to compute confidence intervals for citation shares and prevalence metrics.
  • Distribution-wide rank stability methods (e.g., comparing ranks across samples) quantified how often domain orderings change over repeated samples.
• Main methodological point: treat observed citation shares as sampled statistics from an underlying response distribution and quantify uncertainty via repeated sampling and resampling methods.

Implications for AI Economics

• Measurement & inference: Studies that estimate domain visibility, market share, or referral importance from single runs risk incorrect conclusions. Researchers should treat visibility metrics as stochastic estimators and report confidence intervals computed from repeated samples or bootstrap methods.
• Platform competition & market power: Quantifying platform-mediated visibility (which drives traffic and economic value) requires accounting for stochasticity. Policy assessments (e.g., antitrust, platform gatekeeping) that rely on visibility snapshots may misestimate the extent of concentration or favoritism unless uncertainty is incorporated.
• Business decisions & analytics: Publishers and advertisers using generative search visibility metrics for strategy or ROI estimation should collect repeated samples and report uncertainty; otherwise decisions may respond to sampling noise.
• Methodological guidance for practitioners and researchers:
  • Use repeated sampling (both high-frequency to capture short-term noise and multi-day sampling to capture temporal drift).
  • Compute bootstrap confidence intervals for citation shares and prevalence.
  • Run rank-stability analyses to understand how robust top-N or share-based conclusions are.
  • Determine required sample size empirically: run a pilot to estimate variance, then choose the number of samples needed to achieve the desired confidence-interval width. The paper provides practical guidance on sample sizes conditioned on observed variability.
• Broader implication: The heavy-tailed (power-law) nature of citation distributions implies long tails and high variance, meaning that achieving tight uncertainty bounds can require substantially more sampling than intuition from thin-tailed settings would suggest.