Seventeen ways to pair humans with LLMs: the chosen interaction pattern materially alters recommendations, error types and who bears responsibility; in clinical diagnostics different archetypes produce systematic differences in outputs and reliance, implying nontrivial consequences for productivity, liability and adoption.
Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making
semantic_scholar
descriptive
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
The paper defines 17 reusable human–LLM interaction archetypes and shows that which archetype is adopted meaningfully changes LLM recommendations, error profiles, and downstream decisions in clinical diagnostic cases, with attendant tradeoffs for control, accountability, and information needs.
LLMs are increasingly supporting decision-making across high-stakes domains, requiring critical reflection on the socio-technical factors that shape how humans and LLMs are assigned roles and interact during human-in-the-loop decision-making. This paper introduces the concept of human-LLM archetypes -- defined as re-curring socio-technical interaction patterns that structure the roles of humans and LLMs in collaborative decision-making. We describe 17 human-LLM archetypes derived from a scoping literature review and thematic analysis of 113 LLM-supported decision-making papers. Then, we evaluate these diverse archetypes across real-world clinical diagnostic cases to examine the potential effects of adopting distinct human-LLM archetypes on LLM outputs and decision outcomes. Finally, we present relevant tradeoffs and design choices across human-LLM archetypes, including decision control, social hierarchies, cognitive forcing strategies, and information requirements. Through our analysis, we show that selection of human-LLM interaction archetype can influence LLM outputs and decisions, bringing important risks and considerations for the designers of human-AI decision-making systems
Summary
Main Finding
The paper introduces "human-LLM archetypes" — recurring socio-technical interaction patterns that specify the roles humans and large language models (LLMs) are assigned in human-in-the-loop decision-making. From a scoping review and thematic analysis of 113 LLM-supported decision-making papers the authors derive 17 archetypes, and show (via a case study on clinical diagnostic tasks) that choosing different archetypes meaningfully changes LLM outputs and downstream decision outcomes. The work highlights design tradeoffs (decision control, social hierarchy, cognitive forcing, information requirements) and attendant risks (hallucination, overreliance, sycophancy, bias), arguing that role assignment is a critical determinant of real-world LLM impact beyond benchmark performance.
Key Points
- Definition: A human-LLM archetype captures a socio-technical pattern (role, framing, workflow, social positioning) by which an LLM is integrated into a decision process (e.g., fact-checker, consultant, arbiter, hypothesizer).
- Empirical scope: 113 HITL papers across domains were reviewed and coded to extract recurring patterns; 17 distinct archetypes were identified via thematic analysis.
- Socio-technical drivers: Role assignment depends on problem framing, prompt design/persona, interface and interaction mode, ordering of workflow steps, model fine-tuning, and the human users’ role and authority.
- Measured effects: In clinical diagnostic examples, archetype selection altered LLM outputs on dimensions relevant to decisions (accuracy, inter-agent agreement, information complexity), implying differing risks and benefits.
- Risks & failure modes: Archetypes can amplify hallucination, encourage undue trust/overreliance, activate sycophantic behaviors (model aligning to human framing), and propagate biases depending on how tasks and control are structured.
- Design levers: Authors articulate tradeoffs across (i) decision control (who finalizes decisions), (ii) social hierarchies (how authority is presented), (iii) cognitive forcing (prompts to elicit critical thinking vs. deference), and (iv) information requirements (what context/data are provided).
- Practical contribution: A taxonomy and actionable guidelines to pick and design archetypes appropriate to domain risk and desired human-AI division of labor.
Data & Methods
- Literature search: Scoping review over ACM Digital Library and Web of Science using terms around "LLM" and "decision-making," supplemented via snowball sampling.
- Inclusion criteria: Papers had to involve LLMs in human-in-the-loop decision-making (either user studies where humans interact with LLM outputs or LLM pipelines intended for human review/interaction).
- Corpus: 113 papers passed screening; the authors extracted decision task, domain, intended decision maker, LLM pipeline details (including prompts and system messages where available), and deployment/interaction choices.
- Analysis: Thematic analysis and open coding performed by multiple researchers, iterated through rounds of discussion to derive 17 archetype codes and map examples to each archetype.
- Case study/evaluation: The paper evaluates selected archetypes on real-world clinical diagnostic cases to compare LLM outputs across outcome-relevant variables (accuracy, agreement, information complexity). (Paper presents empirical comparisons; specific model versions and quantitative results are reported in the full text.)
Implications for AI Economics
- Value depends on role design, not just model capability: Economic assessments of LLMs should go beyond model-level benchmarks and account for workplace integration (which archetype is used), since archetype choice changes realized accuracy, productivity, and error-risk.
- Complementarity vs substitution: Different archetypes imply different complementarities with human labor. Archetypes that leave final decision control with humans (consultant, hypothesizer) are more complementary and likely to augment skilled labor; archetypes that automate evaluation/decision (arbiter, grader) create stronger substitution pressures and different distributional impacts.
- Measurement & procurement: Procurement and cost–benefit analyses should include archetype-specific metrics (decision outcome quality, incidence of human override, time-to-decision, audit cost, liability exposure) rather than only model-level scores.
- Incentives, liability, and insurance: Role assignment affects who bears downstream risk (human decision-maker vs system designer). Regulators, insurers, and firms need archetype-aware liability frameworks and insurance products that reflect differing error modes and frequencies across archetypes.
- Market design & product features: Platforms may compete by offering configurable archetypes (role modes) as product features. Pricing, SLAs, and certification could be archetype-differentiated — higher-risk archetypes warrant stricter guarantees and higher monitoring costs.
- Labor market & training: Adoption of certain archetypes changes skill demands (e.g., ability to supervise AI, perform critical evaluation, interpret AI reasoning). Investments in re-skilling should be targeted to the archetypes deployed.
- Policy and governance: Standardization, transparency, and certification efforts should require documentation of deployed archetype(s), prompt/system-message design, and workflow ordering because these materially affect outcomes and externalities.
- Research directions for economists: quantify causal impacts of archetype choice on productivity, error externalities, hiring and wage patterns; model firm decisions over archetype selection when balancing speed, accuracy, liability, and labor costs; estimate social welfare tradeoffs for archetype-driven automation vs augmentation.
Actionable suggestion for applied economic work: when evaluating LLM adoption in a firm or sector, include an archetype mapping step (which human-LLM archetype will be the operational mode?), simulate or pilot different archetypes, and explicitly model monitoring and override costs and liability exposure as archetype-dependent parameters.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — Provides multimethod evidence: a systematic scoping review (113 papers) that grounds a taxonomy, plus empirical tests showing consistent differences across archetypes in a high‑stakes domain; however, the empirical evaluation is domain‑limited (clinical diagnostics), sample size and model/user variation are not broadly representative, and there is no experimental randomization or economic outcome measurement to establish external causal effects on labor or productivity.
Methods Rigormedium — Review and thematic coding appear systematic and comprehensive for the stated scope, and the applied tests use real-world cases to demonstrate sensitivity to interaction design; nevertheless, the empirical component lacks strong identification features (e.g., RCT or natural experiment), details on case/sample size and LLM variation are limited, and qualitative coding may involve subjective judgment without reported interrater reliability in the summary.
SampleScoping review of 113 papers on LLM-supported decision-making (qualitative coding to derive archetypes); empirical evaluation applied those archetypes to a set of real-world clinical diagnostic cases (number of cases and specific LLM(s)/human participants not specified in the summary), comparing archetype-conditioned LLM outputs, error patterns, and human reliance across cases.
Themeshuman_ai_collab labor_markets productivity governance
IdentificationScoping literature review and qualitative thematic analysis to derive 17 interaction ‘archetypes’, plus a comparative empirical evaluation that applies different archetypes to a set of real-world clinical diagnostic cases to observe systematic differences in LLM outputs and downstream decisions; no randomized assignment or formal causal identification strategy.
GeneralizabilityEmpirical tests are confined to clinical diagnostics — high-stakes and domain-specific, so results may not generalize to other sectors (e.g., finance, customer service, manufacturing)., Limited information on the diversity and number of cases, LLM models, and user populations reduces external validity., No randomized or longitudinal design to assess long-run adoption, labor reallocation, or productivity impacts across firms or workers., The archetypes are derived from existing literature and may be biased by publication and domain sampling in the reviewed corpus., Qualitative coding and taxonomy construction may reflect authors' interpretive choices; cultural and institutional contexts could alter archetype behavior.
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The paper introduces the concept of human-LLM archetypes, defined as re-occurring socio-technical interaction patterns that structure the roles of humans and LLMs in collaborative decision-making. Ai Safety And Ethics | null_result | conceptual framework (existence and definition of human-LLM archetypes) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We describe 17 human-LLM archetypes derived from a scoping literature review and thematic analysis of 113 LLM-supported decision-making papers. Ai Safety And Ethics | null_result | number and characterization of human-LLM archetypes (17 archetypes identified) |
Reading fidelity
high
Study strength
medium
|
n=113
17 archetypes identified
|
| We evaluate these diverse archetypes across real-world clinical diagnostic cases to examine the potential effects of adopting distinct human-LLM archetypes on LLM outputs and decision outcomes. Decision Quality | mixed | LLM outputs and decision outcomes in clinical diagnostic cases |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Selection of human-LLM interaction archetype can influence LLM outputs and decisions. Decision Quality | mixed | changes in LLM outputs and decision outcomes associated with different human-LLM archetypes |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Selection of a human-LLM archetype brings important risks and considerations for the designers of human-AI decision-making systems. Ai Safety And Ethics | negative | identified risks and design considerations for system designers |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| The paper presents relevant tradeoffs and design choices across human-LLM archetypes, including decision control, social hierarchies, cognitive forcing strategies, and information requirements. Ai Safety And Ethics | null_result | catalog of tradeoffs and design considerations across archetypes (categories: decision control, social hierarchies, cognitive forcing strategies, information requirements) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| LLMs are increasingly supporting decision-making across high-stakes domains, requiring critical reflection on the socio-technical factors that shape how humans and LLMs are assigned roles and interact during human-in-the-loop decision-making. Ai Safety And Ethics | positive | trend: increased use of LLMs in high-stakes decision-making domains (motivation for study) |
Reading fidelity
medium
Study strength
medium
|
not reported
|