Human-led prompt calibration markedly improves LLM supply‑chain alerts: embedding domain anchors and company context reduces central‑tendency bias and yields better-aligned risk scores with experts, though gains are validated only on a limited pilot sample.

Human-aligned AI for supply chain risk detection: validation and calibration of LLM-based early-warning assessments

Janis Purk · July 27, 2026 · Logistics Research

semantic_scholar quasi_experimental medium evidence 7/10 relevance Summary only summary available; pdf_status=paywall DOI Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

OpenAlex

Latest observation: August 24, 2026

Janis Purk provider ID

Semantic Scholar

Latest observation: August 24, 2026

Janis Purk provider ID

Iterative human-in-the-loop prompt calibration that embeds domain anchor examples and company context reduced midpoint bias and improved MAE and classification alignment between an LLM-based supply-chain early-warning system and expert judgements across two validation rounds.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

OpenAlex · Observed August 24, 2026

View corpus context

0 cumulative citations

Semantic Scholar · Observed August 24, 2026

View corpus context

This paper investigates how a large language model (LLM)-based early-warning system (EWS) for supply chain risk can be systematically aligned with human expert judgement. Building on prior work that introduced the overall system architecture, this study focuses on the detailed prompting logic underlying the automated analysis and on an empirical validation of the resulting AI-based risk assessments. The purpose is to examine how human evaluations can be used to validate and calibrate AI-generated classifications and numerical risk scores, thereby improving their accuracy, reliability and interpretability. In doing so, the paper contributes to a human-centric understanding of AI-supported decision-making in the context of Logistics 5.0. The study draws on two datasets from a real-world validation conducted in January and March 2025, in which AI-generated risk indicators – including impact, likelihood, credibility, sentiment and risk category – were systematically compared with human expert assessments. Deviations between model and expert evaluations were analyzed using established performance metrics such as mean absolute error (MAE), classification accuracy and error distribution patterns. The methodological design follows a human-in-the-loop approach: expert assessments serve as an operational reference standard and are used to iteratively calibrate the prompting logic underlying the automated analysis. Quantitative evaluation is complemented by design-oriented insights derived from the iterative refinement of the system. The findings demonstrate that a structured, iterative human-in-the-loop validation process successfully improves the quality of AI-based risk assessments. In the initial phase, limited contextual grounding led to conservative model behavior, particularly midpoint bias in numerical ratings such as severity, probability and temporal horizon. Prompt refinements systematically integrated supply-chain-relevant anchor examples, partly adapted to company-specific contexts. This enhanced the model's ability to interpret events in operational terms rather than abstract scales. Across the refined prompts, MAE decreased consistently, confirming that context-aware prompt calibration through expert feedback is an effective mechanism for aligning AI outputs with human judgement. The validation is based on two empirical evaluation rounds with a limited number of pilot companies, which constrains immediate generalization across industries and regions. Future research may extend the approach to larger datasets, additional sectors and alternative LLM variants. Nonetheless, the study provides a transferable methodological template for validating and calibrating AI-driven risk detection systems in logistics. More broadly, the findings underline the importance of embedding domain knowledge and expert judgement into AI-based classifications rather than relying solely on probabilistic model behavior, thereby supporting human-centric decision-making in line with Logistics 5.0 principles. The results support logistics managers in assessing when and how AI-generated alerts can be trusted in operational risk monitoring. By documenting the prompting logic and workflow design, the study enables organizations to implement comparable EWSs using low-code environments. The findings show how structured human feedback reduces noise and misclassification, helping firms focus on truly critical disruptions, avoid unnecessary reactive measures and strengthen resilience and sustainability in supply chain operations. By aligning AI-generated assessments with human expertise, the proposed system reinforces human oversight rather than replacing it. This supports transparency, trust and responsible AI use in operational decision-making. In line with Logistics 5.0 principles, the approach reduces cognitive overload caused by information noise and enables more balanced, informed and socially responsible responses to supply chain disruptions. This paper advances research on AI-based supply chain risk monitoring by providing a transparent and empirically grounded analysis of how LLMs can be calibrated through human expert judgement. Beyond presenting the prompting logic of an operational LLM-based EWS, the study demonstrates how structured expert feedback, context-specific anchor examples and iterative prompt refinement systematically improve model behavior across multiple risk dimensions. In contrast to prior conceptual, simulation-based or purely performance-driven studies, this work shows how and why human-in-the-loop calibration reduces systematic biases rather than merely reporting accuracy gains. By explicitly linking prompting logic, expert validation and operational decision logic, the paper contributes a transferable methodological approach and strengthens the practical and theoretical foundations of human-centric Logistics 5.0 risk management.

Summary

Main Finding

A structured, iterative human-in-the-loop calibration of LLM prompting substantially improves the accuracy, reliability and interpretability of an LLM-based early-warning system (EWS) for supply-chain risk. Prompt refinements that embed domain-relevant anchor examples and company-specific context reduced midpoint bias in numerical risk ratings and produced consistent decreases in mean absolute error (MAE) and improved classification alignment with expert judgements.

Key Points

Human-in-the-loop validation: Expert assessments were used as an operational reference standard to iteratively refine the LLM prompting logic rather than only evaluating post-hoc performance.
Midpoint bias discovered: Initial prompts induced conservative, “midpoint” ratings on severity/probability/horizon; lack of contextual grounding drove this bias.
Prompt design matters: Integrating supply-chain-relevant anchor examples and adapting prompts to company context shifted model outputs from abstract scales to operationally meaningful evaluations.
Empirical improvement: Across iterative prompt refinements, MAE decreased and classification accuracy improved; error distributions moved away from central tendency toward better agreement with experts.
Transparency and transferability: The study documents prompting logic and workflow design, enabling implementation in low-code environments and providing a methodological template for other firms.
Limitations: Validation used two empirical rounds (Jan and Mar 2025) with a limited set of pilot companies, constraining immediate generalizability across sectors and regions.

Data & Methods

Datasets: Two real-world validation rounds conducted in January and March 2025; data comprised AI-generated risk indicators for events and parallel human expert assessments.
Risk indicators compared: Numerical scores (impact/severity, likelihood/probability, temporal horizon), credibility, sentiment and categorical risk labels.
Reference standard: Expert judgements served as the operational ground truth for calibration and evaluation.
Evaluation metrics: Mean absolute error (MAE) for numerical ratings, classification accuracy for categorical labels, and analysis of error distribution patterns (e.g., central tendency / midpoint bias).
Iterative procedure: Prompt refinements were made in cycles—experts reviewed model outputs, highlighted systematic deviations, and prompts were adjusted to include anchor examples and contextual cues; subsequent rounds measured improvement.
Complementary analysis: Design-oriented insights recorded from the iterative refinement process to inform workflow and low-code implementation.

Implications for AI Economics

Value of human capital: Expert feedback raises the marginal value of AI outputs by improving precision and interpretability. This implies a trade-off between upfront expert labor (training/calibration) and downstream operational savings (fewer false positives/negatives, better-targeted responses).
Cost-benefit and ROI: Reducing noise and misclassification can lower unnecessary reactive measures, shrink disruption-related costs, and improve supply-chain resilience—improving ROI for EWS investments. However, firms must account for the recurring cost of expert-in-the-loop calibration as contexts, products, and suppliers change.
Adoption and trust: Demonstrable alignment with human judgement increases managerial trust and likelihood of operational adoption. Trustworthiness and explainability can accelerate diffusion of EWS solutions across firms that are otherwise reluctant to act on black-box alerts.
Market structure and services: A market opportunity exists for calibrated, domain-adapted EWS offerings (including low-code kits, custom prompt libraries, and expert calibration services). Vendors may differentiate on pre-built anchor libraries and sector-specific prompting templates.
Scalability and generalizability: Human calibration improves local accuracy but limits plug-and-play scalability—each company or sector may need bespoke prompt anchoring. Economies of scale will depend on how reusable anchor examples and calibration workflows are across customers.
Labor and skill implications: Operations and risk teams will need new skills (prompt engineering, oversight, contextual anchoring) and possibly dedicated roles for continuous calibration and governance.
Regulatory and governance effects: Embedding expert oversight supports compliance and auditability of AI-driven operational decisions. Documented prompting logic and validation workflows can reduce legal and reputational risk when regulators require human oversight or explainability.
Model choice and robustness: The study suggests value in context-aware prompt design irrespective of specific LLM variants, but economic decisions should consider differences in model costs, latency, and performance under domain shift—affecting platform/provider selection.
Research and investment priorities: Further investments into larger cross-sector validations, standardized anchor libraries, and automation-assisted calibration (semi-automated expert labeling tools) can improve scalability and reduce marginal calibration costs.

Overall, the paper implies that economically efficient deployment of LLM-based EWSs balances investment in human expertise for calibration against the operational cost-savings from better-targeted alerts—favoring human-centric, domain-anchored approaches for high-stakes supply-chain risk monitoring.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The study uses real-world data and expert judgements as an operational reference standard and reports consistent improvements (MAE reductions, better classification alignment) across two rounds, which supports the claim that prompt refinements improved performance; however, the design lacks randomization or external controls, sample size and scope are limited (few pilot companies), and potential confounders or temporal effects are not ruled out, limiting causal confidence and generalizability. Methods Rigormedium — Strengths: systematic iterative procedure, clear evaluation metrics (MAE, classification accuracy), use of domain experts as reference standard, and analysis of error-distribution shifts (midpoint bias). Weaknesses: no randomized or counterfactual design, limited rounds and pilot sample, unclear statistical testing or power, potential overfitting to the expert panel and company contexts, and limited external validation across models/sectors. SampleTwo real-world validation rounds conducted in January and March 2025 using a limited set of pilot companies (number unspecified); data comprised LLM-generated risk indicators for events (numerical scores for impact/severity, likelihood/probability, temporal horizon; credibility; sentiment; categorical risk labels) paired with parallel human expert assessments used as the reference standard. Themeshuman_ai_collab adoption org_design productivity governance IdentificationBefore-and-after iterative intervention: LLM prompting was iteratively refined using expert feedback and improvements in model outputs were measured across two validation rounds (January and March 2025) by comparing AI-generated risk indicators to parallel expert judgements; no randomization or external control group was reported. GeneralizabilityLimited number of pilot companies—unclear sample size and sectoral coverage, Only two validation rounds over a short time window (Jan–Mar 2025) — potential temporal/context dependence, Relies on company-specific context and anchor examples, which may not transfer across firms, products, or regions, Performance may vary with different LLM models, versions, or prompt implementations, Expert judgements used as ground truth may reflect local biases and are not an objective external benchmark

Claims (8)

Claim	Direction	Outcome	Confidence & Evidence	Details
Structured, iterative human-in-the-loop calibration of LLM prompting improved the accuracy, reliability, and interpretability of an LLM-based early-warning system for supply-chain risk. Output Quality	positive	Agreement and reliability of LLM-generated supply-chain risk assessments	Reading fidelity high Study strength medium	not reported 0.48
Initial prompts induced conservative midpoint ratings for severity, probability, and temporal horizon, reflecting a lack of contextual grounding. Decision Quality	negative	Numerical risk ratings for impact/severity, likelihood/probability, and temporal horizon	Reading fidelity high Study strength medium	not reported 0.48
Embedding supply-chain-relevant anchor examples and company-specific context shifted LLM outputs from abstract scales toward more operationally meaningful evaluations. Decision Quality	positive	Operational relevance and alignment of LLM-generated risk evaluations	Reading fidelity high Study strength medium	not reported 0.48
Across iterative prompt refinements, mean absolute error decreased and classification accuracy improved relative to expert judgements. Error Rate	positive	Numerical-rating error and categorical classification alignment with expert assessments	Reading fidelity high Study strength medium	not reported 0.48
Expert assessments were used as an operational reference standard to iteratively refine prompting logic, rather than only to evaluate performance after the fact. Governance And Regulation	positive	Human oversight and calibration of AI-generated risk assessments	Reading fidelity high Study strength medium	not reported 0.48
The study's immediate generalizability is constrained because validation relied on two empirical rounds and a limited set of pilot companies. Other	negative	External validity and cross-sector or cross-region generalizability	Reading fidelity high Study strength high	not reported 0.8
Human calibration can improve local accuracy but limits plug-and-play scalability because companies or sectors may require bespoke prompt anchoring. Adoption Rate	mixed	Scalability and transferability of calibrated early-warning systems	Reading fidelity high Study strength low	not reported 0.24
Documented prompting logic and validation workflows can support compliance and auditability of AI-driven operational decisions. Regulatory Compliance	positive	Auditability and compliance support for AI-driven operational decisions	Reading fidelity high Study strength speculative	not reported 0.08