Language models do not produce fixed measurements: sampling, hidden updates, floating-point effects and expert routing make LLM outputs variable and able to change empirical results; researchers should record model identifiers, timestamps, seeds when available, and provide multiple draws and replication materials so analyses can be assessed and reproduced.

Main Finding

Large language model (LLM) outputs are inherently stochastic for multiple reasons beyond the commonly discussed sampling temperature. Because model outputs are draws from a distribution — not fixed measurements — researchers must treat, report, and analyze LLM-generated data accordingly. Exact reproduction is generally impossible when using hosted proprietary APIs; even open-weight local runs retain nontrivial sources of nondeterminism (hardware, libraries, parallelism). The paper documents the sources of randomness, demonstrates their empirical impact via sentiment classification of corporate filings and downstream regressions, and proposes a reporting standard and practical guidance for authors and data editors.

Key Points

• LLM output variation arises from several distinct mechanisms:
  • Deliberate sampling (temperature, top-k/top-p, seed). Setting temperature = 0 (greedy decoding) reduces sampling randomness when exposed, but many providers now hide or manage temperature.
  • Silent model updates: providers can change models, system messages, routing, or guardrails over time without changing API names, breaking reruns.
  • Floating-point rounding and parallel numerical nondeterminism: batching, hardware differences (GPU type/count), and scheduling can flip near-tied logits and change generated tokens even under zero temperature.
  • Expert routing / mixture-of-experts: routing decisions can depend on batch composition, so concurrent requests can affect outputs.
• Researchers have varying degrees of control:
  • Controllable: prompts, explicit decoding parameters (if provided), multiple independent model draws, archiving raw outputs and prompts.
  • Partly controllable or not controllable: underlying model weights, serving infra, provider-side sampling policies, hardware specifics for hosted APIs.
• Empirical consequences:
  • The authors illustrate through sentiment classification of annual corporate filings that repeated LLM calls can produce different labels, which in turn can change regression coefficients and significance in downstream analyses.
  • Aggregation (averaging or majority voting across multiple independent runs) improves stability and consistency of measurements.
• Policy and practice recommendation:
  • Treat LLM outputs as stochastic: report them as distributions, not point measurements.
  • Authors should archive prompts, raw model outputs, model identifiers, timestamps, provider fingerprints, and decoding parameters; run multiple independent draws and report variability.
  • Data editors should require extended metadata for LLM-generated data and adapt reproducibility checks to account for nondeterminism.

Data & Methods

• Literature/usage survey:
  • Sample: 145 high-ranked journals (AJG rank 4 or 4*) over January 2020–July 2026 (105,464 publications).
  • Detection: searched OpenAlex abstracts for mentions of 212 LLM names and general LLM terms (“LLM”, “generative AI”, etc.).
  • Findings: sharp rise in LLM mentions; by July 2026 ~6% of papers in these journals had LLM-related keywords in title/abstract (conservative estimate). Closed-weight models (OpenAI, Google, etc.) dominate usage; many providers do not expose temperature settings.
• Empirical experiment (illustrative):
  • Task: sentiment classification of annual corporate filings (common use-case where model output becomes data).
  • Approach: repeated model calls to produce labels/scores; examined variation across runs and measured consequences in downstream regressions (how coefficients/significance change when using different draws).
  • Results: repeated runs yield nontrivial label variation; downstream regression estimates and significance can change; aggregating multiple draws reduces instability.
• Related evidence cited:
  • Prior studies documenting instability (Ouyang et al. 2024; He 2025; Wang & Wang 2025; Li et al. 2025; Atil et al. 2024) and demonstrating that aggregation across runs can improve consistency.

Implications for AI Economics

• Measurement error and inference:
  • LLM-induced variability introduces an additional, often underreported source of measurement error that can bias estimates, change standard errors and p-values, and affect identification strategies when model outputs are treated as covariates, outcomes, or sample selectors.
  • Single-run analyses risk reporting results that are contingent on an unobserved draw from a model’s output distribution.
• Best-practice methods to mitigate risks:
  • Multiple independent draws: obtain many replicate outputs per observation and report distributional summaries (means, medians, variances), confidence intervals, or bootstrap-style uncertainty over model draws.
  • Aggregation/ensembling: use majority vote, averaged continuous scores, or other aggregation rules to stabilize measurements.
  • Sensitivity analysis: show robustness of main results across independent runs, across different models (when possible), and to alternative aggregation rules.
  • Prefer open-weight/local execution for greater control where reproducibility is critical, but still archive hardware/software stack info since floating-point and parallelism effects remain.
  • When possible, set and report deterministic decoding (temperature = 0) and seeds — but report caveats: seeds only help when the full stack is controlled and provider guarantees hold.
• Reporting and reproducibility requirements for economic research:
  • Researchers should include in papers and replication packages: prompts (exact text), full raw outputs from each run, model identifier returned by the API, date/time of calls, provider/endpoint, all decoding parameters (temperature, top-k/top-p, seed), number of independent draws per observation, aggregation rule(s), and any available provider fingerprints/version metadata.
  • Data editors and journals should update policies to treat LLM-generated data as probabilistic: require metadata, multiple-run evidence, and archiving of raw outputs; accept probabilistic replication (reproducing reported distributions/predictive variability) rather than exact bitwise reproduction when exact reproduction is impossible.
• Broader implications:
  • Many published results relying on single-run LLM-generated variables may be fragile; the research community should re-evaluate key findings where model-generated measurements are central.
  • As closed providers increasingly manage sampling parameters server-side, researchers will have less ability to eliminate randomness — increasing the importance of reporting, replication with multiple draws, and alternative measurement designs.

Practical checklist (compact) for authors to include with LLM-generated data: - Exact prompt text(s) and any preprocessing rules. - Raw model outputs for every run (not only aggregated labels). - Model identifier(s) returned by API, provider endpoint, date/time of calls. - Decoding parameters: temperature, top-k, top-p, seed (if exposed). - Number of independent draws per input and aggregation rule(s). - Notes on whether model weights are open/closed and whether run was local or hosted; if local, record hardware, OS, library versions, and precision settings. - Statement acknowledging nondeterminism and sensitivity analyses demonstrating robustness.

Bottom line: in AI-driven empirical work, treat LLM outputs as stochastic measurements, document them rigorously, quantify and report their variability, and adapt inference and reproducibility standards to reflect these realities.