Universal toxicity detectors miss roughly a third of disability-specific harms in AI-generated images; targeted guidelines and lightweight model adaptation help, but remain well short of the accuracy achieved on general toxicity detection.

Main Finding

State-of-the-art, one-size-fits-all toxicity detectors and vision–language models systematically fail to detect community-specific representational harms in text-to-image (T2I) outputs. In two disability communities (blind/low vision and dwarfism), ~32–38% of images that general detectors mark “safe” are judged harmful by community experts. Community-specific adaptation (in-context prompting, VQA, or small-data fine-tuning) helps substantially but remains far below the performance achieved for general-purpose toxicity detection, highlighting both the urgency and difficulty of deploying inclusive safety systems.

Key Points

• Scope and claim:
  • The paper introduces Community-Specific Toxicity Detection (CTD): detectors tailored to harms defined by particular communities rather than universal, aggregated safety taxonomies.
  • CTD targets representational harms (e.g., unrealistic assistive devices, infantilization, fantasy/entertainment tropes) that often do not appear in general toxicity datasets.
• Empirical gap:
  • Using community-developed guidelines, 32.2% (BLV) and 37.5% (DWF) of T2I images labeled safe by SoTA detectors violate community-specific guidelines.
• Models & zero-shot performance:
  • Dataset evaluated across multiple T2I models (GPT-Image-1, Imagen4-Ultra, Stable Diffusion 3.5) and many VLMs.
  • In zero-shot setting (models only given community guidelines at inference), most general-purpose VLMs and safety-specialized detectors score near-zero F1 on CTD (examples: many models F1 ≈ 0; GPT-4o F1 0.35 on BLV; GPT-4o-mini F1 0.51 on DWF; ShieldGemma2 F1 ~0.27–0.31).
  • Baselines: Random baseline F1 ≈ 0.321 (BLV) and 0.375 (DWF); retrieval (nearest neighbor CLIP) F1 ≈ 0.449 (BLV) and 0.695 (DWF).
• Adaptation helps but is insufficient:
  • Prompt-based adaptation (in-context learning, VQA) using large VLMs (e.g., GPT-4o) improved F1 to ~0.50 (BLV) and ~0.78 (DWF).
  • Parameter-efficient fine-tuning (PEFT) on smaller VLMs (0.5B–7B) with <100 community-specific examples raised F1 to ~0.48 and ~0.59 (best cases).
  • Even after adaptation, CTD performance is well below ~0.9 F1 typical for general-purpose toxicity detection.
• Practical challenges:
  • CTD requires community-specific guidelines that evolve as models change; detectors are sensitive to guideline shifts and may need frequent retraining.
  • Data scarcity: only small numbers of annotated generated images per community are typically available, so sample-efficient methods are essential.
  • Scaling: naive aggregation across communities is inappropriate because harms are particularistic; supporting many evolving, non-aggregable guidelines raises architecture and operational questions.
• Research directions articulated:
  • Learning from scarcity; adapting to changing guidelines; system architectures for many evolving guidelines; infrastructures for distributed community–researcher collaboration and trust-building.

Data & Methods

• Communities and guidelines:
  • Two disability communities: blind/low vision (BLV) and people with dwarfism (DWF).
  • Experts identified 6 representational-harm categories per community (e.g., non-anatomical distortions, inappropriate eye coverings, infantilization, portrayal as fantasy characters). Guidelines validated with advocacy organizations.
• Dataset:
  • 400 community-authored prompts per community (reflecting authentic depictions).
  • For each prompt, images sampled from three T2I models → 1,200 images per community → total 2,400 images.
• Annotation:
  • Five disability experts labeled images for all applicable harms; an image is unsafe if any harm present.
  • Inter-rater: Cohen’s kappa on a random sample (3 raters) = 0.65 (BLV) and 0.97 (DWF). Full dataset annotated at one expert per image after reliability checks.
• Rationale-assisted pipeline:
  • GPT-4o used to generate and filter natural-language rationales; only examples where GPT-4o judgments aligned with human experts were retained for rationale pairing.
  • Training sets constructed to be balanced across harm categories: up to 10 single-harm examples per harm, producing 90 training examples for BLV and 40 for DWF (paired with GPT-4o rationales).
  • Validation: prompts shared with training but not same images; Test: remaining images (954 BLV, 1,067 DWF).
• Models evaluated:
  • Closed-source large VLMs (GPT-5 series, GPT-4o, etc.), open-source VLMs (0.5B–11B param), and safety-specialized detectors trained on universal taxonomies (LlavaGuard, QwenGuard, ShieldGemma2).
• Evaluation:
  • Binary safe/unsafe classification (unsafe = positive class); Precision, Recall, F1 reported.
  • Judgment–rationale consistency measured by extracting harms from model rationales using GPT-4o (extractor validated with >97% exact match).
• Adaptation approaches:
  • Zero-shot (guidelines only at inference).
  • Prompt-based adaptation: in-context learning (ICL), VQA prompting (rationale + instruction).
  • Parameter-efficient fine-tuning with small (<100) community-specific datasets.
• Key quantitative outcomes (representative):
  • Zero-shot: many VLMs F1 ≈ 0; GPT-4o F1 0.35 (BLV) / 0.19 (DWF); GPT-4o-mini 0.24 (BLV) / 0.51 (DWF).
  • Adaptation: GPT-4o with ICL/VQA reached F1 ≈ 0.50 (BLV) and 0.78 (DWF).
  • PEFT on smaller models (0.5–7B) achieved best F1 ≈ 0.48 (BLV) and 0.59 (DWF) with <100 examples.
  • SoTA general-purpose detection for mainstream harms: F1 ≈ 0.9 (benchmark for comparison).

Implications for AI Economics

• Externalities and welfare:
  • Missed representational harms impose non-trivial social and economic costs on marginalized groups (psychological harm, reputational damage, reduced access to opportunities). These externalities are currently not internalized by models and platforms using one-size-fits-all detectors.
• Compliance, liability, and regulatory risk:
  • Regulators and civil-society stakeholders increasingly expect AI systems to mitigate harm to vulnerable groups. Failure to detect community-specific harms may raise legal and reputational risk for firms, prompting potentially costly remediation or sanctions.
• Cost structure and scaling:
  • CTD requires ongoing community engagement, expert-labeled examples, and frequent adaptation as model behaviors and harm categories evolve. This introduces recurrent fixed and variable costs:
    • Labor costs: recruiting and compensating domain experts and advocacy organizations for guideline design, annotation, and validation.
    • Operational costs: storing/serving many community-specific models/guidelines or running prompt-based adapters at inference.
    • Retraining/updating costs: sensitivity to guideline changes implies repeated fine-tuning or frequent prompt engineering.
  • These costs can create market opportunities for specialized vendors or platform-level services offering managed CTD modules, but also raise entry barriers for smaller product teams.
• Product design and segmentation:
  • Platforms may need to move from a single safety pipeline to configurable, per-user or per-community safety modes. This supports better user alignment but increases product complexity and may fragment moderation policies across jurisdictions and customer segments.
• Market competition and value capture:
  • Firms that invest early in robust CTD (infrastructure, community partnerships, modular detectors) can differentiate on trust, compliance, and user safety—potentially monetizable through enterprise safety SLAs or API tiers.
  • Conversely, costs may push some providers to conservative content restrictions (overblocking), creating welfare losses (reduced utility) for users who want authentic depictions.
• Policy and public good implications:
  • There is a role for shared, public-good resources (community-validated guideline repositories, shared annotated datasets, or funded partnerships) to reduce duplicated costs across firms and ensure equitable coverage across many small or under-resourced communities.
  • Subsidies or regulation encouraging open standards for CTD would lower coordination costs and help internalize externalities.
• Strategic research & investment priorities:
  • Investing in sample-efficient adaptation methods, robust rationale-grounding, and modular architectures (supporting many evolving taxonomies without retraining full stacks) has high expected economic return given the regulatory and reputational stakes.
  • Funding long-term community partnerships is essential; technical fixes alone will not scale fairness or legitimacy.

Limitations noted by authors (relevant for economic assessment): - Study covers two disability communities; other communities may surface different harms and scaling challenges. - CTD performance improvements still lag general toxicity detection—continued R&D needed. - Effective CTD depends on trustable engagement with community representatives, which has non-trivial time and coordination costs.

Takeaway for economists and decision-makers: universal safety detectors miss substantial, measurable harms for specific communities. Addressing these gaps requires ongoing investment (labor, annotation, infrastructure), new business models (specialized detection services, per-community safety tiers), and public-good coordination to contain costs and ensure inclusive mitigation of model-driven externalities.