A multi-agent LLM tool (PKAI) helps business process analysts produce more accurate and useful conceptual models in controlled tests and demonstrates promise in a real-world deployment; however, evidence rests on a quasi-experiment and a single field case, limiting broad claims about productivity gains.

Main Finding

The paper develops and empirically validates PKAI, a novel LLM-based multi-agent system that operationalizes the full process-knowledge acquisition pipeline (preparation, socialization, externalization). Grounded in knowledge-acquisition theory and practitioner input, the authors derive 19 design requirements and show—via expert demonstrations, a quasi-experimental study, and a real-world illustrative case—that PKAI improves the semantic and pragmatic quality of conceptual process models, enables asynchronous and more autonomous elicitation by domain experts, and standardizes parts of the knowledge-acquisition workflow. The study also documents limitations of current LLMs (hallucinations, syntactic/consistency issues) and argues for human oversight and targeted engineering (RAG, structured outputs, prompt design) to mitigate risks.

Key Points

• Research gap and framing
  • Reframes business-process discovery as a knowledge-acquisition task with two central stages from Nonaka/Cooke: socialization (eliciting tacit knowledge) and externalization (formalizing into conceptual models).
  • Identifies limitations of purely manual elicitation and of automated discovery approaches; motivates LLM augmentation.
• Contributions
  • Problem diagnosis and 19 design requirements for LLM support in process knowledge acquisition, grounded in theory and interviews with practitioners.
  • PKAI: an instantiated, LLM-based multi-agent system that assigns specialized agents to preparation, socialization, and externalization tasks and uses LLM-enhancing techniques (prompt engineering, structured outputs, retrieval-augmentation).
  • Empirical evaluation across three rounds:
  • Demonstration and validation with business process analysts (11 interviews; mapping of requirements; follow-up validation with 9 analysts).
  • A quasi-experimental study comparing process analysts with and without PKAI support—showing improved semantic and pragmatic dimensions of conceptual models (3QM framework).
  • An illustrative real-world case showing applicability under business complexity and impact on the acquisition process.
• Practical observations about LLM use
  • LLMs can guide interactive elicitation (follow-ups, clarifications) and pre-formalize knowledge into model-ready textual structures.
  • Structured outputs and retrieval-augmented generation are central to reducing volatility and improving relevance.
  • LLMs are more effective as augmenting tools that standardize and scale elicitation rather than as autonomous replacements—human analysts remain necessary for validation and final formalization.
• Limitations noted
  • Persistent risks of hallucination, syntactic inconsistencies, and unpredictable model behavior requiring oversight.
  • Existing LLM-generated models may still fall short of experienced analysts on some quality dimensions, though support improves outcomes on semantic/pragmatic fronts.

Data & Methods

• Methodological approach
  • Design Science Research (DSR) with iterative cycles: problem diagnosis → artifact design/instantiation → empirical evaluation.
  • Grounding in knowledge-acquisition theory (Cooke, Nonaka) and BPM literature.
• Data sources & empirical work
  • Practitioner interviews: in-depth interviews with 11 experienced process analysts to elicit real-world challenges and validate design requirements; 9 of these participated in follow-up demonstrations mapping requirements to the PKAI instantiation.
  • Artifact design: PKAI multi-agent system built using state-of-the-art LLM techniques (prompt engineering, structured outputs like JSON/BPMN-compatible structures, retrieval-augmented generation) and agent orchestration to operationalize preparation, socialization, and externalization.
  • Evaluation:
    • Demonstration/validation with practitioners to confirm requirement mapping and functionality.
    • Quasi-experimental study assessing model quality using the 3QM framework (semantic, pragmatic, syntactic dimensions), finding statistically meaningful improvements in semantic and pragmatic quality when analysts were supported by PKAI.
    • An illustrative real-world case study showing applicability under complex business conditions and qualitative impacts on the knowledge-acquisition workflow.
• Measures & frameworks
  • Process-model quality evaluated via the 3QM framework (semantic, pragmatic, syntactic), with emphasis in results on semantic and pragmatic gains.
  • Design requirements synthesized from literature and practitioner input (19 items spanning socialization and externalization needs).

Implications for AI Economics

• Productivity and task reallocation
  • PKAI-style tools can raise the productivity of business process analysts by automating preparatory research, structuring elicitation, and pre-formalizing models—shifting analyst time from repetitive elicitation toward validation, interpretation, and higher-value modeling tasks.
  • Domain experts can perform more autonomous, asynchronous elicitation, reducing coordination costs and analyst labor intensity; this creates complementarities rather than pure substitution in many cases.
• Labor-market and human-capital effects
  • Demand shifts toward skills in supervising, validating, prompt/design engineering, and integrating LLM outputs—implying upskilling needs and possible wage-premia for oversight roles.
  • Lower barriers to entry for smaller firms to perform sophisticated process discovery could alter demand for specialized consulting and outsourcing services in BPM.
• Cost, adoption, and diffusion
  • Standardization and automation of elicitation reduce per-project transaction and time costs, improving ROI for investments in process-improvement initiatives and incentivizing wider adoption of BPM tools.
  • However, governance, validation, and mitigation of LLM errors introduce additional oversight and compliance costs; net economic gains depend on the balance between efficiency improvements and these assurance costs.
• Market structure and competition
  • Firms able to safely and effectively integrate organization-specific documentation into retrieval-augmented pipelines may gain competitive advantage via proprietary process-knowledge assets encoded in augmented models.
  • A market for verticalized LLM-enabled BPM platforms and specialist services (integration, prompt engineering, model auditing) is likely to expand.
• Measurement and evaluation needs for economists
  • Need for quantitative cost–benefit analyses: time saved in elicitation, error rates affecting downstream operations, and impacts on process performance metrics.
  • Research agenda: randomized controlled trials or larger-scale field experiments to quantify productivity, error/repair costs from LLM hallucinations, effects on organizational outcomes (cycle time, compliance, error rates).
• Policy and risk considerations
  • Data-privacy and IP implications from feeding proprietary process documentation into LLM pipelines affect adoption and competitive dynamics; regulation or vendor constraints could change cost structures.
  • Liability and accountability for automated or semi-automated process models require governance frameworks, which have economic implications (insurance, compliance spending).

Suggested next empirical steps for AI economics researchers - Conduct larger-scale randomized field experiments measuring time savings, model rework/repair costs, and downstream process performance when using PKAI-like tools. - Model adoption decisions at firm level incorporating oversight costs, privacy constraints, and heterogeneous analyst skill levels to estimate aggregate productivity gains. - Study labor-market impacts: tasks reallocation, wage effects, and demand for new roles (prompt engineers, LLM auditors) in BPM-related occupations.