AI assistants can silently shrink engineers' safety reasoning and amplify failure risk unless workflows are redesigned; the same AI tool can either improve or degrade safety analysis depending on collaboration structure, so regulators and firms should qualify workflows, not just software.

Main Finding

AI assistance in safety engineering can either improve or substantially degrade safety analysis depending on collaboration design. The paper introduces the "competence shadow": AI outputs systematically narrow human reasoning along multiple cognitive and organizational mechanisms. These mechanisms compound multiplicatively (not additively), so naïve efficiency gains from AI (faster drafts, lower apparent labor) can mask large, long-tail reductions in hazard identification. Therefore the central policy is: AI in safety engineering is a collaboration-design problem (workflow qualification), not merely a software procurement or tool qualification decision.

Key Points

• Safety competence is multidimensional and resists single-number benchmarking. The paper models competence as a vector C = ⟨D, S, E, Cx, J⟩:
  • D: domain knowledge
  • S: standards expertise
  • E: operational experience
  • Cx: contextual understanding
  • J: judgment (risk calibration)
• Three fundamental barriers to benchmarking safety AI:
  • Context-dependent ground truth (severity depends on deployment context)
  • Inherent incompleteness (some failure modes reveal only after deployment/incidents)
  • Legitimate expert disagreement (multiple valid decompositions and perspectives)
• Competence shadow: AI’s partial competence profile does not just produce omissions in its output — it systematically prevents humans from hypothesizing and retaining certain hazards.
• Four cognitive/organizational mechanisms create the shadow:
• Scope framing (αframe): AI’s ontology/framings anchor hypotheses.
• Attention allocation bias (β): reviewers spend more time verifying AI output than independently exploring.
• Confidence asymmetry (ηdisagree): humans under-retain findings that contradict the AI.
• Organizational time compression (γ): management shortens analysis timelines when AI appears fast, reducing human effort and amplifying the other mechanisms.
• These mechanisms compound multiplicatively; the paper defines an effective anchoring coefficient αeff = αframe · β · ηdisagree.
• Four canonical collaboration structures with different shadow exposures:
  • π1 Serial Dependency (AI-first, human reviews): all shadow mechanisms active; highest risk.
  • π2 Independent Analysis & Synthesis (humans and AI analyze independently, then synthesize): eliminates framing, attention, and confidence asymmetry structurally; only time compression remains.
  • π3 Tool Augmentation (AI confined to auxiliary tasks like formatting/standards cross-reference): shadow absent for core reasoning if clean boundaries are enforced; risk from boundary-errors (ε, δ).
  • π4 Human-Initiated Exploration (human does clean-room analysis, then asks AI for gaps): eliminates framing and attention biases; confidence asymmetry and time compression may remain.
• Quantitative framing: quality Q = |Sidentified|/|S| with baseline identification probabilities qh and qAI. Under plausible illustrative parameter values (e.g., αframe≈0.8, exploration retention ≈0.3, ηdisagree≈0.7, γ≈0.6), multiplicative effects can produce much larger degradations than additive intuition suggests.
• Practical takeaway: tool capability is insufficient criterion; deployment structure (who sees what when, and incentives) determines safety outcome.

Data & Methods

• Conceptual + formal theory paper rather than an empirical trial.
• Methods:
  • Formal modeling of safety competence as a 5-dimensional vector.
  • Identification of four cognitive/organizational parameters (αframe, β, ηdisagree, γ) grounded in automation-bias and anchoring literature.
  • Definition of four canonical human-AI collaboration structures (π1–π4) that differ in information flow and which shadow mechanisms are active.
  • Derivation of closed-form performance bounds (qualitatively summarized here) for each structure, expressing final identification quality Q in terms of qh, qAI, and the shadow parameters. Key analytic result: shadow factors multiply across mechanisms producing an effective anchoring coefficient αeff.
• Calibration / illustrative parameter choices are drawn from related literature (automation bias, human-AI teaming studies) and recent industry capability studies on LLMs in hazard-analysis tasks; the paper notes these are illustrative and calls for empirical measurement of shadow parameters.
• Paper situates theory against empirical findings showing LLMs can produce useful hazard analyses but are inconsistent and often lack depth in operational/contextual dimensions.

Implications for AI Economics

• Procurement vs. Workflow Qualification:
  • Economic value of safety-AI tools cannot be judged by capability benchmarks alone. Buyers must value and pay for workflow design that resists competence shadow (e.g., independent analyses, human-initiated workflows), not just the model license or API.
• Productivity metrics and measurement:
  • Short-term productivity (drafts per hour) overstates value because it ignores compounded reductions in hazard coverage and tail risk. Cost-benefit analyses must include the expected marginal increase in residual risk caused by the competence shadow and potential incident/recall liabilities.
• Incentives and agency problems:
  • Management incentives to compress schedules (γ) can create negative externalities—apparent savings from faster certification may raise expected incident costs later. Contracting and compensation should internalize long-run safety risk (e.g., holdbacks, longer warranty/recall liabilities, or insurance loads).
• Market structure and product differentiation:
  • Demand will emerge for (a) tools and services that enable shadow-resilient workflows (supporting π2, π3, π4 patterns), and (b) third-party “workflow qualification” and audit services. Vendors may command premium pricing for interfaces & governance features that demonstrably reduce αeff and β.
• Insurance, liability, and regulation:
  • Insurers and regulators should treat AI assistance as altering organizational process risk; underwriting and certification regimes should require evidence of workflow qualification (independent analyses, bounded AI roles, audits). Standards should move from tool qualification to explicit workflow requirements.
• Evaluation and research economics:
  • Economists and empirical researchers should measure shadow parameters (αframe, β, ηdisagree, γ) via controlled experiments, A/B tests, and retrospective incident analyses. Sample-size calculations must account for low base rates and long-tail incidents—expected-value losses can be dominated by rare but severe events.
• Policy levers:
  • Mandate or incentivize independent-analysis structures for safety-critical certification; require disclosure of AI roles in safety artifacts; subsidize creation of incident-sharing databases (to reduce incompleteness) to better evaluate long-tail risks.

Practical short list for economic actors evaluating safety-AI: - Prioritize collaboration structures over raw model benchmarks; prefer π2/π3/π4 patterns when certification and long-term safety matter. - Incorporate multiplicative shadow factors into expected-loss models and pricing of AI-enabled productivity. - Require third-party workflow audits, and align contractual liability and insurance with residual risk after workflow mitigation.