Dermatology AI fails mainly from missing disease coverage, not skin tone: cancer-trained models collapse on non-neoplastic conditions while dermatology-pretrained foundation models retain transferable representations, and modest local labeling (∼10 examples per category) largely recovers usable performance.

Main Finding

Disease-distribution shift (training on cancer-focused images → deployment on non-neoplastic conditions) drives much more of the dermatology-AI generalization gap than skin-tone underrepresentation in the datasets studied. The gap reflects a representational collapse in cancer-specialized models, whereas dermatology foundation models retain more transferable structure and can be cheaply adapted (≈10 labeled examples per local category) to recover most attainable performance.

Key Points

• Distribution shift >> skin-tone effect:
  • Cancer baseline (ResNet-50 fine-tuned on HAM10000+ISIC) balanced accuracy: 0.62 in-domain → 0.21 on SCIN (tone-diverse, non-neoplastic).
  • Tone-stratified test (DDI) produced much smaller within-disease tone gaps (0.10–0.18) that were inconsistent in direction across models.
• Representational deficit, not just missing labels:
  • kNN neighbor-purity lift (label-free measure of separability) in-domain: cancer baseline +0.42; on SCIN it collapses to +0.06 (near chance).
  • Dermatology-pretrained models retain much more structure on unfamiliar conditions (e.g., DermLIP SCIN lift +0.23).
• Cheap adaptation is effective when representations are good:
  • Adaptation with ~10 labeled examples per category recovered most attainable performance for dermatology foundation models (few-shot probe often matching/exceeding full linear probe).
  • Representation quality (purity lift) strongly tracks recoverable performance (correlation r ≈ 0.90 across the four encoders evaluated; descriptive given small n).
• Models evaluated (frozen feature extractors): cancer baseline (ResNet-50), DermLIP, MONET, DINOv3 (general-purpose vision model).
• Reproducibility and low compute: experiments used only public datasets, free-tier cloud compute, and code/repro protocol released by authors.

Data & Methods

• Datasets:
  • HAM10000 + ISIC 2019 (35,346 images): pooled, neoplastic, used as in-domain/source training for the cancer baseline.
  • DDI (656 images): tone-stratified, used to isolate skin-tone effect (binary malignant/benign).
  • SCIN (6,517 images after cleaning; tone-diverse, dominated by non-neoplastic): used to isolate distribution shift.
• Models: frozen feature extractors — ResNet-50 cancer baseline (2048-d), DermLIP (512-d), MONET (1024-d), DINOv3 (768-d).
• Evaluation protocol (identical downstream pipeline across encoders):
  • Linear probing (L2-regularized logistic regression) reporting balanced accuracy with patient-level splits.
  • Label-free representation quality: k-NN neighbor-purity lift (k=10, cosine metric), reported as purity minus chance-floor to adjust for imbalance.
  • Low-compute adaptation: full probe vs few-shot probe (10 examples/class, 5 draws) on SCIN categories; feature standardization tested.
• Key quantitative results:
  • Cancer baseline: in-domain balanced accuracy 0.62 → SCIN 0.21; DDI tone gap 0.14 (95% CI 0.03–0.28).
  • DermLIP: in-domain 0.52 → SCIN 0.36; DDI tone gap 0.14 (0.03–0.26); SCIN purity lift +0.23.
  • MONET: in-domain 0.52 → SCIN 0.34; DDI tone gap 0.10 (0.02–0.20); SCIN purity lift +0.14.
  • DINOv3: in-domain 0.47 → SCIN 0.31; DDI tone gap 0.18 (0.05–0.31); SCIN purity lift +0.16.
• Limitations noted by authors:
  • DDI and SCIN are US-sourced proxies, not genuinely resource-constrained settings (RCS) cohorts — conclusions for RCS are inferential.
  • Small sample sizes for some Fitzpatrick groups in DDI (statistical power limited).
  • Only frozen-feature evaluation; fine-tuning/parameter-efficient tuning might change results.
  • Architectural and pretraining differences confound some comparisons; embedding dimensionality differs across models.

Implications for AI Economics

1. Resource allocation priorities
2. Prioritize broad disease-distribution coverage (pretraining or curation of datasets reflecting target burden) over collecting tone-only examples if the deployment gap is driven by missing disease types. Collecting tone-diverse examples is still necessary for fairness auditing but may not close deployment gaps caused by label-space mismatch.

3. Investing in a stronger starting representation (dermatology foundation models trained on varied clinical conditions) yields higher marginal returns for downstream adaptation than investing the same resources into local retraining from a narrowly specialized model.

4. Cost-benefit of data collection vs model selection

5. Low-cost, small local labeling (≈10 examples per category) can recover substantial performance when starting from an appropriate foundation model — implies a cheaper adaptation pathway for clinics/RCS than large-scale local dataset collection or full model retraining.

6. The one-time cost of extracting frozen features (computationally modest; feasible on CPU for downstream probing) and a small labeled collection may dominate marginal economic costs, making local adaptation financially tractable.

7. Market and policy implications

8. Demand signals: clinics and health systems in RCS may prefer access to broadly pretrained dermatology foundation models plus support for small-scale local labeling/adaptation rather than numerous narrow-specialist classifiers.
9. Incentives for model providers: commercial or public providers should prioritize breadth of pretraining across clinically relevant conditions and release adaptable, well-documented encoders to maximize deployment value.

10. Regulatory and procurement guidance: regulators and purchasers should assess representation quality across disease distributions (not only demographic parity) when evaluating models for deployment in different clinical contexts.

11. Efficiency and equity trade-offs

12. Misallocating scarce data/annotation budgets toward tone-only collection (absent disease-coverage adjustments) risks underdelivering improvement where it matters most for clinical utility in different settings.

13. However, fairness auditing remains essential: tone-diverse datasets are required to detect and mitigate demographic harms. The economic prescription is therefore twofold — invest in disease-diverse representations to raise baseline utility, and maintain tone-diverse auditing to ensure equitable outcomes.

14. Limitations to economic generalization

15. The studied datasets are proxies; deployment economics in actual RCS could differ (different disease mixes, image capture variability, patient demographics, and implementation constraints).
16. The benefit estimates (e.g., ≈10 labels/class sufficing) are empirical to these models/datasets; local validation and cost-estimation remain necessary before scaling procurement or funding decisions.

Actionable takeaways for decision-makers - When constrained, acquire or license a broadly pretrained dermatology foundation model rather than repurposing a narrow cancer-specialist classifier. - Budget for a small, targeted local labeling effort (order of tens of labeled examples per clinically relevant category) for adaptation rather than large-scale relabeling. - Require representation-quality metrics (e.g., label-free separability across target conditions) as part of model evaluation for procurement, not just in-domain accuracy. - Continue to fund tone-diverse data collection for auditing and fairness mitigation, but align primary investment to reflect disease-distribution coverage needed for target deployments.

Repository and reproducibility - The authors released code and evaluation protocol: https://github.com/Nirajan995/dermatology-generalization-gap

Summary judgment - From an AI-economics perspective, the paper argues that investing in broad, clinically aligned pretraining and modest local labeling yields superior value for adapting dermatology AI to new settings, whereas focusing only on skin-tone diversification will likely underdeliver when the underlying disease distribution differs.