Imperfect LLM detectors can backfire — by changing incentives they may push people to use LLMs more and even lower output quality. The authors demonstrate this with a strategic user model and reproduce a characteristic 'rise-then-fall' pattern in arXiv abstracts' word frequencies.

Main Finding

Introducing imperfect LLM detectors can produce counterintuitive, perverse downstream effects because users respond strategically. In a stylized model combining human–LLM collaboration and post-processing (gaming), the authors show that detectors can (1) increase LLM usage for some users, (2) decrease overall output quality even when the detector targets a quality-reducing attribute, and (3) reliably produce a “rise-then-fall” pattern in the detectored attribute — a pattern the authors reproduce empirically for word-frequency signals on arXiv abstracts.

Key Points

• Intervention view: LLM detectors are interventions that change incentives and thereby affect not only the flagged attribute but also LLM usage and multi-dimensional output quality.
• Stylized mechanism: Outputs are D-dimensional; the detector uses a threshold on one coordinate (z1) while quality depends on all coordinates. Users choose:
  • α ∈ [0,1], fraction of task outsourced to the LLM (mt(α) = αℓt + (1−α)ht);
  • post-processing z′ ≤ mt(α) (reducing coordinates) at separable costs C.
• User utility trades off quality Q(z), detection penalty β·1[z1>ν], production cost (γ·(1−α)), and post-processing cost C.
• Main theoretical results:
  • Theorem 1 (sufficient condition): There exist plausible parameter regimes where adding a detector (finite threshold and penalty) induces some users to increase α relative to the no-detector baseline. Intuition: users will post-process away the detector-targeted harm, letting them exploit LLM strengths in other dimensions.
  • Theorem 2 (necessary conditions): The paper characterizes when such increases in α are impossible; in particular, if quality increases with the detected attribute, the increase cannot occur.
  • Theorem 3: There are parameter regimes where adding a detector reduces users’ equilibrium output quality, even when the detector targets a quality-reducing coordinate. Intuition: detection can induce under-use of LLMs and thereby lose complementary quality improvements in non-detected dimensions.
  • Theorem 4: The detected attribute exhibits a clean “rise-then-fall” pattern when users gain access to LLMs and then face detectors. This pattern matches empirical observations (e.g., certain word frequencies rising after ChatGPT release and falling later).
• Robustness and constructions: The authors give explicit constructions (2‑D examples) and sufficient parametric forms of Q and C to show these effects are achievable; assumptions on C (A1–A5) ensure post-processing is costly and well-behaved.
• Empirical component: Beyond theory and stylized simulations (Figures 1), the paper documents and reproduces the rise-then-fall pattern for detected word/style signals across arXiv abstracts and shows that these patterns are more prevalent in the LLM era than in prior windows over the past decade.

Data & Methods

• Analytic model:
  • D-dimensional output space; detector = threshold classifier on dimension 1: F(z)=1[z1>ν].
  • Quality Q(z) separable across coordinates, concave components Qi; Q1 may be positively or negatively correlated with quality.
  • User types t = (ht, ℓt, γt): human output, LLM output, and human production cost; assume ℓt1 > ht1 (LLM has larger detector signal).
  • Post-processing modeled via separable cost C(z, z′) where z′≤z; two regimes: finite-cost C or infinite-cost C∞ (no post-processing).
  • Users maximize Ut = Q(z) − β·1[z1>ν] − γt(1−α) − C(mt(α), z).
  • Existence and characterization of optima proven; tie-breaking rules specified.
• Theoretical analysis:
  • Constructive proofs to exhibit types and parameter regimes yielding the three main phenomena. Lemmas show monotonicities of after-post-processing utility and behavior under extreme penalties.
  • Theorems give sufficient and necessary conditions; illustrative 2-D constructions used for intuition.
• Simulations/figures:
  • Stylized numerical plots (Figure 1) illustrating how α, Q, and detected attribute change under detector introduction under example parameterizations.
• Empirical validation:
  • Textual analysis on arXiv abstracts: track frequencies of candidate detector signals (words / stylistic markers) over time; demonstrate rise after public LLM releases and fall after detector-aware behavioral responses.
  • The authors report that the rise-then-fall pattern is more prevalent in the LLM era than in comparable prior windows (details in paper Figures and appendix).

Implications for AI Economics

• Detectors are policy interventions with endogenous behavioral responses. Economic analysis of LLM governance must treat detectors as incentive-changing mechanisms, not passive measurements.
• Perverse incentives:
  • Institutions deploying detectors to reduce LLM use may unintentionally increase use for some agents because post-processing can neutralize flagged harms while preserving LLM benefits. This complicates policy goals like discouraging LLM dependence.
  • Quality trade-offs: detectors that target a single superficial signal can reduce aggregate quality when they push users away from using LLMs in complementary (non-detected) dimensions.
• Measurement and inference:
  • Observed rise-then-fall patterns in detector signals (e.g., specific words) are consistent with strategic adaptation, so simple frequency trends are an ambiguous signal of true LLM prevalence or quality trends.
• Design recommendations for practitioners and policymakers:
  • Anticipate strategic responses: model or experiment with how users will post-process and reallocate work before deploying automated detectors.
  • Target multi-dimensional quality outcomes rather than single superficial features; evaluate detectors by downstream outcomes (actual quality, welfare) not just hit rates on a proxy feature.
  • Reduce false positives and ensure robustness to trivial gaming to avoid costly post-processing and misdirected behavior.
  • Consider complementary or alternative interventions: outcome-based incentives, auditing, transparency and ex post verification, or mechanisms that change the benefits/costs of delegation (e.g., training, regulated disclosure) rather than only penalizing signals.
• Research implications:
  • The paper links strategic classification literature with human–AI collaboration, highlighting the need for models of endogenous delegation and gaming together.
  • Future empirical work should measure welfare-relevant outcomes (quality, productivity) and heterogeneity across user types; policy evaluations should consider equilibrium behavioral changes.
• Limitations to keep in mind:
  • The model is stylized (threshold detector on a single coordinate, separable quality), and empirical validation is limited to aggregate signal patterns (arXiv word frequencies). Real-world detectors, multi-dimensional detection strategies, user learning over time, and network effects can change quantitative outcomes. Nonetheless, the qualitative warnings about perverse incentives are robust.

Overall, the paper provides theoretical and empirical evidence that LLM detectors — when imperfect and focused on narrow signals — can distort incentives in ways that undermine both the intended reduction in LLM usage and the intended improvement in output quality. Policymakers and institutions should treat detection as an active intervention and evaluate detectors by their downstream equilibrium effects.