0 cumulative citations
View corpus contextAn LLM-powered tender-analysis tool cuts initial screening time for public procurement while keeping retrieval accuracy high. The system shifts employee effort from reading documents to orchestrating and verifying AI-generated outputs, suggesting LLMs can materially change knowledge-work processes.
Rethinking Knowledge Work: Designing LLM-based Systems for Complexity Management
openalex
descriptive
medium evidence
8/10 relevance
Summary only
summary available; pdf_status=paywall
DOI
Source
PDF
An LLM-based document-analysis tool, developed via design-science with an SME, reduced initial screening time for public tenders while maintaining high retrieval accuracy in evaluations on real tenders.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Knowledge workers increasingly face complexity in managing heterogeneous documents, strict compliance requirements, and cross-functional processes, which disproportionately burden small and medium-sized enterprises (SMEs). Particularly in public procurement, tender analysis involves processing heterogeneous documents, complying with requirements, and coordinating across functions to inform bid decisions. Using design science research with an SME, we investigate how to manage complexity in document-intensive knowledge work—exemplified by public tender analysis—across document, compliance, and process dimensions, deriving four design requirements and two design principles. We instantiate these in a Large Language Model (LLM)-based artifact that supports document analysis. Evaluated on real tenders, the artifact reduces initial screening time while maintaining high retrieval accuracy. Our findings demonstrate that knowledge work can be advanced by LLMs, reducing complexity and transforming the nature of human work—shifting the focus from reading and extracting information to orchestrating and verifying LLM-generated outputs.
Summary
Main Finding
An LLM-based artifact, designed via design science research with an SME, can meaningfully reduce the time required for initial screening of document-intensive public tenders while preserving high retrieval accuracy. More broadly, LLMs can reduce complexity in document-heavy knowledge work and shift human roles from reading/extracting information toward orchestrating and verifying model outputs.
Key Points
- Problem context: Knowledge workers (especially in SMEs) face high complexity from heterogeneous documents, strict compliance requirements, and cross-functional coordination; public procurement/tender analysis exemplifies this.
- Complexity dimensions addressed: document heterogeneity, compliance constraints, and process coordination.
- Research approach: design science research conducted in collaboration with an SME to derive practical design guidance.
- Design outcomes: four design requirements and two design principles were derived (targeting document, compliance, and process dimensions).
- Technical instantiation: the requirements/principles were implemented as an LLM-based artifact to support automated document analysis in tender workflows.
- Evaluation results: on real tenders, the artifact reduced initial screening time while maintaining high retrieval accuracy.
- Work transformation: human experts’ tasks shift from manual extraction to orchestrating, validating, and integrating LLM-generated outputs.
Data & Methods
- Methodological framework: design science research (iterative artifact design, build, and evaluation) in partnership with an SME stakeholder to ensure ecological validity.
- Data: real public tender documents used for artifact evaluation (heterogeneous document formats and content typical of procurement processes).
- Metrics reported: time for initial screening and retrieval accuracy (exact numerical values not provided in the summary).
- Evaluation: empirical testing of the LLM-based artifact on actual tender cases to measure efficiency gains and accuracy preservation.
- Note on reproducibility: the summary does not specify the exact LLM architecture, retrieval/augmentation techniques, preprocessing steps, or dataset size—those would be required for full replication.
Implications for AI Economics
- Productivity and cost effects:
- Reduces transaction and search costs in document-intensive tasks (notably for SMEs), potentially lowering barriers to entry in public procurement markets.
- Time savings in initial screening translate to lower labor costs per bid and higher throughput of tender evaluations.
- Labor reallocation and skill demand:
- Shifts worker tasks from routine reading/extraction toward higher-level orchestration, validation, and exception handling—raising demand for skills in prompt engineering, model oversight, and compliance interpretation.
- Potentially increases the productivity of existing staff rather than replaces them entirely, particularly where verification and legal accountability remain necessary.
- Market structure and competition:
- Lowering the operational burden of tender analysis may increase SME participation in procurement, affecting competitive dynamics and procurement outcomes.
- Demand for commercial LLM-based workflow tools is likely to grow, creating new product markets and vendor opportunities targeting SMEs.
- Regulatory and compliance considerations:
- While LLMs can aid compliance by surfacing relevant requirements, reliance on model outputs raises risks (errors, hallucinations, lack of auditability). Firms and regulators may need standards for verification, provenance, and record-keeping.
- Measurement challenges for economic impact:
- Accurate assessment requires measuring net effects including verification overhead, error correction costs, and impacts on bid success rates.
- Distributional effects matter: gains may accrue unevenly across firms depending on digital maturity and ability to integrate LLM tools.
- Policy implications:
- Support for SME adoption (training, subsidized tools) could amplify competition in procurement markets.
- Policymakers should consider guidance on AI use in regulated workflows, emphasizing transparency and accountability to avoid adverse compliance outcomes.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports empirical evaluation on real tender documents and shows measurable gains (reduced screening time, maintained retrieval accuracy), but evidence comes from a design-science engagement with a single SME, likely a modest sample, no randomization or counterfactual control, and limited external validation, which constrains causal claims and external validity.
Methods Rigormedium — Methods combine systematic design-science development with implementation of an LLM-based artifact and empirical evaluation using task-based metrics; however, rigor is limited by likely small/selected sample from one partner SME, lack of pre-registered hypotheses, absence of randomized or quasi-experimental identification, and limited robustness checks or sensitivity analyses.
SampleEvaluations used a set of real public procurement tender documents provided by the collaborating SME, with heterogeneous formats and content representative of the firm's workload; human evaluators measured initial screening time and retrieval accuracy comparing conventional manual screening to the LLM-assisted artifact (sample size and precise selection criteria not specified).
Themeshuman_ai_collab productivity
GeneralizabilitySingle-SME case study limits external validity to other firms or sectors, Focused on public procurement tenders; applicability to other types of knowledge work is untested, Unclear language/jurisdiction dependence (e.g., procurement rules, document languages), Performance may depend on the specific LLM model, prompts, and tool implementation used, Small or non-random sample of tenders may bias observed effects
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Knowledge workers increasingly face complexity in managing heterogeneous documents, strict compliance requirements, and cross-functional processes, which disproportionately burden small and medium-sized enterprises (SMEs). Organizational Efficiency | negative | administrative/operational burden from document heterogeneity, compliance, and cross-functional coordination |
Reading fidelity
high
Study strength
medium
|
n=1
|
| In public procurement, tender analysis involves processing heterogeneous documents, complying with requirements, and coordinating across functions to inform bid decisions. Task Allocation | null_result | process steps and task complexity involved in tender analysis (document processing, compliance checking, cross-functional coordination) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Using design science research with an SME, we derive four design requirements and two design principles for managing complexity in document‑intensive knowledge work. Organizational Efficiency | positive | number and content of design requirements and design principles derived |
Reading fidelity
high
Study strength
medium
|
n=1
|
| We instantiate these requirements and principles in an LLM-based artifact that supports document analysis. Organizational Efficiency | positive | existence and functionality of an LLM-based document analysis artifact |
Reading fidelity
high
Study strength
medium
|
n=1
|
| Evaluated on real tenders, the artifact reduces initial screening time while maintaining high retrieval accuracy. Task Completion Time | positive | initial screening time (task completion time) and retrieval accuracy (relevance/quality of retrieved information) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Knowledge work can be advanced by LLMs, reducing complexity and transforming the nature of human work—shifting the focus from reading and extracting information to orchestrating and verifying LLM-generated outputs. Task Allocation | positive | shift in task composition/role of humans in knowledge work (from extraction to orchestration/verification) |
Reading fidelity
high
Study strength
medium
|
n=1
|