Accuracy alone is not enough: to deserve professional deference, generative AI must signal its limits, let users inspect and contest outputs, and recognise users as legitimate knowers; without these three conditions, even 'accurate' systems can induce unjustified reliance and harm.

Main Finding

The paper develops a normative, constitutive framework of "epistemic trustworthiness" for generative AI used in high‑stakes professional workflows. It argues that, beyond standard output measures (accuracy, calibration, fairness, explainability), warranted reliance on a GenAI output requires three jointly necessary and non‑fungible conditions at the user–system interaction: (1) epistemic humility, (2) epistemic access, and (3) resistance to epistemic injustice. Only when these hold (together with first‑order competence) does reliance become epistemically warranted.

Key Points

• Motivation: Generative AI produces fluent, open‑ended, testimony‑like outputs that professionals may incorporate into their reasoning. Behavioral alignment with correct outputs or output‑centric metrics alone do not guarantee that user reliance is epistemically justified.
• Three constitutive conditions:
• Epistemic humility — the system must represent and communicate its limits of competence in actionable ways (not merely generic disclaimers). Subcomponents: (H1) actionable limitation‑signalling (what users should do given uncertainty) and (H2) interactional humility (sustained, responsive limitation signalling across exchanges).
• Epistemic access — users must be practically able to inspect, interpret, question, and contest the evidential basis of an output in context (beyond providing citations or opaque explanations).
• Resistance to epistemic injustice — systems must recognise users and affected communities as legitimate epistemic agents, avoiding marginalisation of their concepts, language, or experiences.
• These three conditions are derived from social epistemology: second‑order competence maps to humility; audience‑orientation splits into practical communicative access and recognitional respect.
• The framework treats trustworthiness as relational and interactional (a property of the user–system relation at the moment of reliance), not merely a property of outputs or governance arrangements.
• It is normative (principled) rather than a plug‑and‑play measurement instrument; operationalising thresholds requires context and empirical work.
• The authors give layer‑explicit diagnostics (model, data, interface) to locate where breakdowns occur and apply the framework to case studies in legal reasoning, medical reasoning, and hiring to show harms that accuracy/fairness/usability metrics can miss (e.g., confidently fabricated citations, unflagged out‑of‑scope medical recommendations, marginalisation of applicant knowledge).

Data & Methods

• Methodological approach: conceptual and normative theory-building grounded in social epistemology (competence and audience‑orientation) and translated to the GenAI sociotechnical stack using an instrumental stance (do not attribute genuine beliefs to models; evaluate whether outputs function epistemically).
• Evidence and illustrations: literature synthesis from HCI, explainable AI, social epistemology, and responsible AI; layer‑explicit analysis (model/data/interface) and qualitative case analyses of real‑world scenarios in law, medicine and hiring that demonstrate diagnostic value of the framework.
• Scope and assumptions: focused on GenAI deployed in contexts where human epistemic agency is at stake (high‑stakes professional settings); presupposes first‑order competence (accuracy/reliability) as necessary but insufficient; deliberately normative rather than fully operationalised.
• Not an empirical experimental paper—no new large datasets or randomized trials; calls for context‑sensitive operationalisation and empirical validation of the normative conditions.

Implications for AI Economics

• Market valuation and product differentiation
  • Trustworthiness becomes a multidimensional good: firms that deliver epistemically trustworthy systems (humility + access + recognition) can charge premiums or obtain preferential procurement, distinct from those who only optimize accuracy/fluency.
  • Vendors face new design/QA costs (verification interfaces, provenance infrastructure, interactional humility mechanisms, participatory design to avoid epistemic injustice), affecting product pricing and margins.
• Adoption, productivity, and labor markets
  • Productivity gains from GenAI depend on whether users can reliably integrate outputs into decisions. Lack of epistemic trustworthiness increases verification/friction costs, reducing net productivity and adoption speed in high‑stakes domains.
  • Human labor complementarities shift: skills that enable inspection, contestation, and contextual judgement (domain expertise, critical evaluation, source literacy) become more valuable; routine tasks may be automated only where epistemic access and humility are sufficient.
• Information asymmetries and transaction costs
  • Standard accuracy signals are insufficient to resolve information asymmetries between vendors and institutional buyers. Epistemic access and humility function as additional trust signals; absent them, buyers incur higher transaction costs (audits, third‑party evaluations, legal liability buffers).
  • Market failures may arise if purchasers cannot distinguish systems that merely appear competent (fluent/confident) from those that provide warranted reliance; this can create adverse selection pressure.
• Liability, insurance, and regulatory economics
  • Procurement and regulatory frameworks may begin to condition deployment on meeting epistemic trustworthiness criteria (e.g., interface requirements for inspectability, standards for limitation signalling, participatory design requirements to prevent epistemic injustice). Compliance costs and certification markets will grow.
  • Liability allocation and insurance pricing will need to factor in whether a system provided adequate epistemic access/humility; insurers and regulators may discount coverage for systems that fail these interactional requirements.
• Externalities and distributional impacts
  • Epistemic injustice risks translate into economic inequities: marginalized groups may be systematically disadvantaged (hiring, healthcare, legal outcomes), producing social costs and reduced labor market efficiency.
  • Deployments that appear accurate on aggregate could still produce concentrated harms (misrecognition, exclusion), suggesting need for targeted evaluation and redistribution or remediation policies.
• R&D and public goods
  • Investments in infrastructures that support epistemic access (provenance, retrieval pipelines, UI affordances for contestability) can have public‑good characteristics; underinvestment is likely without policy incentives or standards.
  • Standardisation and third‑party auditing markets (for humility/access/recognition features) can emerge, affecting industry structure.
• Measurement & economic evaluation
  • Cost–benefit analyses of GenAI in organizations must incorporate the downstream costs of verification, contestation, and reparative processes when epistemic trustworthiness is absent.
  • Economic models of AI adoption should extend beyond accuracy to include parameters for verification friction, trustworthiness premiums, and the value of avoided epistemic harms.
• Policy levers
  • Procurement standards, sectoral regulation, liability rules, and subsidy programs for trustworthy infrastructure can realign incentives toward designs that enable epistemically warranted reliance.
  • Encouraging participatory design and community recognition mechanisms addresses distributional inefficiencies and internalises social costs of epistemic injustice.

Overall, the paper suggests that economic assessments and policy for GenAI should treat epistemic trustworthiness as an explicit, measurable dimension of product quality and social welfare — one that shapes adoption, pricing, labor complementarities, liability, and distributional outcomes in ways that accuracy‑centric metrics do not capture.