Kaggle medals remain informative but perishable: almost all predictive power sits in the first year, and the apparent collapse of upload-based credentials after the platform phased out upload evaluations is mainly a consequence of a frozen medal stock aging, not clear evidence that individuals became less skilled under AI.

Main Finding

Generative-AI did not make credentials on Kaggle worthless. Medals remain informative about subsequent (hidden-test) performance, but their informativeness is highly perishable (mostly concentrated in the first year), institution-bound (depends on the evaluation format that awarded them), and interdependent with the rest of a participant’s record. The large collapse in the value of upload-earned medals after the platform shifted to code-execution evaluation is driven mostly by institutional “stranding” (the upload format being phased out and the medal stock aging), not by a uniform collapse of personal skill signals. Simple changes in how credentials are read (isolated badge-count vs full-profile) materially change apparent revaluations. Platform lifetime tiers also throw away nontrivial signal (13–16%).

Key Points

• Data scope and outcome

  • Audit of Kaggle’s full archive (2010–2026) focusing on 444,698 participations; two concurrent evaluation formats: upload-competitions (submit predictions on published data) and code-competitions (platform runs code on hidden data).
  • Outcome measured: one minus final hidden-test percentile (higher = better).

• Freshness and decay

  • Nearly all predictive power of a medal is in its first year. Pre-AI fresh medals (<1 year) have slopes ≈ 0.15–0.18 per log medal (unconditional) and under full-profile conditioning deliver the vast majority of explained within-competition variance (two under-one-year bands alone explain ~99% pre-AI, ~95% AI-era).
  • Medals older than 1 year carry very little incremental informativeness.

• Institutional stranding explains most of the upload-medal collapse

  • Before AI: upload- and code-earned medal stocks were similarly informative (stock slopes ≈ 0.067 and 0.065 per log medal, conditional on one another and prior participation).
  • After the AI transition (competitions with deadlines 2023+; 2022Q4 treated as transition and excluded), upload-stock slope fell by 0.055 (SE 0.007, P < 0.001) from 0.067 to 0.012 — an 82% loss. Code-stock rose by 0.022 (to 0.087, P = 0.04).
  • Decomposition of the upload decline (total −0.055): composition (age-mix shift) −0.028; curve shift (band slopes change) −0.015; interaction +0.007; aggregation error −0.019. Aging (composition + plausible shares of interaction/aggregation) explains roughly half to three quarters of the decline (competition-cluster bootstrap: 43–62% under conservative attribution, 64–87% under generous attribution).
  • Interpretation: many upload medals were stranded because the platform largely stopped issuing new upload medals before or during the AI transition; the existing stock aged per the pre-existing decay pattern.

• Reading lens and proxying effects

  • Two reading lenses: badge-count read (credential in isolation) vs full-profile read (what a medal adds conditional on the rest of the record).
  • Old upload-earned medals appear more informative under a badge-count read in the AI-era, but this inflation disappears under a full-profile read — stale medals proxy for other parts of the holder’s profile (e.g., recent code medals, experience).
  • Old code-earned medals show the opposite sign-change between lenses: full-profile weights for older code medals rise in the AI era because other, previously stronger overlapping signals (upload medals) lost predictive power; this is re-weighting, not intrinsic durability.

• Person-level patterns

  • Predicted person-level shifts toward an “AI-like working style” were not found. A behavioral index (submissions per competition; share same-day resubmissions; strength of first scored submission) meant to capture AI-like fewer-iteration high-quality first shots predicts performance similarly in code- and upload-competitions.
  • Under person and competition fixed effects, the index’s difference in returns between code and upload is +0.006 per index unit (SE 0.005, P = 0.28) — statistically indistinguishable from zero.

• Platform tiers and alternative aggregation

  • Kaggle’s official lifetime tiers (Expert, Master, Grandmaster) discard ~13–16% of the information in medal histories.
  • An index that weights recent medals more heavily, estimated on pre-AI data only, outperforms the official tiers at predicting AI-era performance.

• Robustness and auxiliary findings

  • Community competitions (which award no tier medals) show the same upload fall and code rise (upload −0.046, code +0.074; n = 344,181), supporting that the stock divergence is not an artifact of medal competitions alone.
  • Analyses are associational: results describe informativeness of records under the measurement system, not causal claims about individual skill changes or the amount of AI use.

Data & Methods

• Data

  • Kaggle competition archive 2010–2026; primary estimation samples split into pre-AI-era (competition deadlines through 2022Q3) and AI-era (deadlines 2023 onward); transition quarter 2022Q4 excluded.
  • 444,698 participant–competition observations in main analyses (teams credited individually when competing jointly).
  • Medal counts by format, medal-age bands (0–1y, 1–2y, 2–3y, 3–5y, 5y+), prior participation counts, submission telemetry.

• Outcome and predictors

  • Outcome: one minus final percentile on hidden-test leaderboard (task-specific, objective).
  • Predictors: log medal counts by format and age band; total medal stocks; behavioral index (submissions per competition, share same-day resubmits, strength of first scored submission).

• Estimation

  • Participation-level regressions with competition fixed effects (comparison among entrants in the same competition).
  • Standard errors two-way clustered by user and competition.
  • Decomposition of stock slope changes into composition (age-mix), curve shift (band slopes), interaction, and aggregation error; competition-cluster bootstrap for confidence intervals on decomposition shares.
  • Reading-lens comparisons: unconditional (badge-count) vs full-profile conditioning (control for other-format medals and prior participation); balanced panel checks (users present in both eras) used to rule out survivorship as primary driver of some patterns.
  • Person-level tests use person and competition fixed effects to isolate within-person changes.

• Identification and limits

  • The study measures informativeness (predictive power for hidden-test outcomes) — a measurement-system property. It does not observe or identify AI usage by individuals, nor does it make causal claims about changes in underlying skills.

Implications for AI Economics

• Credentials remain useful but are perishable and institution-dependent

  • Credential value depends heavily on recent, validated outputs. Markets that rely on lifetime, aggregated, or frozen credentials risk mispricing candidate ability after institutional evaluation changes or technological shifts.
  • Policy and organizational screening should prioritize recent validated evidence (e.g., executed code, proctored assessments, or time-bound demonstrations) over distant historical badges.

• Institutional design matters

  • Much of the apparent devaluation after an AI-era transition can stem from institutional choices (which evaluation formats are used, whether credentials are replenished). Design features that preserve ongoing validation (e.g., requiring execution on hidden data, maintaining active credentialing streams) sustain credential informativeness.
  • Platforms and credentialing bodies should consider mechanisms that allow credentials to be refreshed, decay-aware tiering, or explicit dating/expiration of signals.

• Reading rules and hiring practice

  • Employers and platforms should read credentials in profile context, not in isolation. Simple badge-count heuristics can mislead because stale badges can proxy for other attributes and over- or under-state helpful information.
  • Aggregation rules (like lifetime tiers) should be revisited: weighting recent performance more heavily recovers substantial predictive power.

• Labor-market dynamics and inequality

  • Short-lived signals favor agents who can continually produce validated evidence (ongoing contest participation, up-to-date portfolios). This could advantage incumbents with resources to continually replenish credentials and disadvantage those whose prior achievements are frozen.
  • Credential stranding is a socio-economic problem: as evaluation standards change (for reasons technological or institutional), holders of previously useful credentials may be asset-stranded unless institutions provide transition paths.

• Generalizability, measurement, and future research

  • The audit is an end-to-end measurement in one high-quality, objective setting (Kaggle), but institutional structures there are shared broadly (objective scoring, public credentials, screening). Results suggest broader lessons but should be tested across occupations and credential systems (degrees, certifications, platform reputation).
  • Future work should examine policy designs for credential replenishment, expiry, or revalidation; the labor supply responses to short-lived signals; and causal links between AI usage, skill acquisition, and credential informativeness.

Limitations to keep in mind: the analysis is associational (no direct observation of AI assistance), restricted to task performance on Kaggle-style benchmarks, and focuses on measurement/informativeness rather than causal effects on human skill or welfare.