Cognition-inspired LLM agents that guide users step-by-step turned fragmented knowledge into more usable system outputs and raised workers' efficiency ratings in a single firm pilot; the results suggest AI can scaffold complex cognitive work, but evidence is preliminary and single-site.

Main Finding

Embedding cognition‑inspired, scaffolding‑style guidance into AI agents (via expert cognitive models used as prompts for large language models) can provide structured, process‑oriented support for practitioners performing system tasks. A prototype tested in a software implementation firm produced more complete and professional task outputs and raised users’ efficiency ratings, suggesting intelligent scaffolding can help convert “messages in bottles” into usable system knowledge.

Key Points

• Problem: Many knowledge‑transfer outputs fail to be applied because practitioners lack system cognition — the ability to acquire, process, and apply information within a specific system context. Existing AI/tools tend to give fragmented cues or ready solutions rather than process‑oriented guidance.
• Conceptual approach: The study adapts the educational idea of scaffolding (progressive, structured support to enable learning) and links it with cognition‑inspired modelling to design tools that guide practitioners through system cognitive processing.
• Design artifacts:
  • System scaffolding structure model — a representation of the process and cognitive steps needed for system tasks.
  • Intelligent scaffolding enactment process — how the scaffold is applied during task execution.
• AI implementation: Expert cognitive models are converted into prompts for large language models to create AI agents that enact the scaffolding. These agents were embedded in a prototype system (agent‑based scaffolding).
• Evaluation results: In a software implementation firm, the prototype:
  • Helped users produce more structured, complete, and professional outputs.
  • Received favorable efficiency ratings from practitioners.
  • Provided preliminary evidence (non‑experimental) that the scaffolding approach supports system cognitive processing.
• Limitations noted by authors: Preliminary evaluation in a single organizational setting; evidence is suggestive rather than causal. Further testing needed for generalizability and long‑term effects.

Data & Methods

• Research method: Design Science Research (DSR) — iterative artifact design, implementation, and evaluation aimed at producing usable design knowledge.
• Artifact development: Combined literature on scaffolding and cognition with expert cognitive modelling to define scaffolding structure and enactment processes.
• AI/technical method: Expert cognitive modelling outputs translated into prompts for large language models to build agentized scaffolding; embedded in a prototype software tool.
• Evaluation: Field implementation in a software implementation firm. Assessment appears to include qualitative observations and user efficiency ratings; reported outcomes include task output quality improvements and favorable user perceptions.
• Evidence level: Prototype evaluation with preliminary, context‑specific findings; not a controlled experiment or large‑N study.

Implications for AI Economics

• Productivity & returns to AI investment:
  • Agentized scaffolding can increase the effective productivity of knowledge workers by structuring cognitive work, potentially raising output quality and speed for complex system tasks.
  • By improving conversion of documented knowledge into usable practice, such tools could raise the ROI of prior knowledge‑creation investments (reducing wasted “messages in bottles”).
• Task composition and labor demand:
  • Shifts in required skills: greater emphasis on supervisory, judgment, and system‑design skills as routine system cognition is partly automated; reduced demand for purely procedural cognition.
  • Potential for task reallocation within firms — junior or less experienced workers may handle more complex system tasks with scaffolding, changing wage structures and career ladders.
• Diffusion and scaling of expertise:
  • Scaffolding agents encode expert cognitive processes, enabling scaling of scarce expert knowledge across staff and locations — a channel for faster knowledge diffusion within firms and across adopters.
  • This could lower frictions to adopting complex systems and services, affecting market structure in sectors reliant on system implementation (e.g., enterprise software, IT services).
• Complementarity with human capital:
  • Tool effectiveness depends on quality of expert cognitive models and organizational integration; returns are complementary to investments in expert modeling and process design.
  • Potential to raise productivity non‑uniformly across firms (those that invest in modeling/embedding scaffolds may gain a competitive advantage), with implications for firm‑level dispersion in productivity.
• Measurement and policy considerations:
  • Empirical assessment of AI’s productivity contributions should account for process‑oriented tools (not only solution outputs) and measure effects on task quality, learning, and diffusion of expertise.
  • Workforce policies and training should consider upskilling to integrate, supervise, and refine scaffolding agents.
• Risks & open questions affecting economic impact:
  • Generalizability: single‑firm evidence; broader economic impact depends on replication across industries and sustained usage.
  • Quality of encoded expertise: erroneous or biased expert models could propagate systematic errors at scale.
  • Labor market dynamics: potential displacement of intermediate tasks, changing bargaining power and wage effects that require empirical tracking.

Overall, the paper provides design knowledge showing a path for AI to augment cognitive, process‑oriented aspects of knowledge work — a channel that can increase the productivity and scalability of complex system tasks and has consequential implications for firm performance, labor demand, and the diffusion of expertise.