An adaptable, evidence-focused AI companion integrated into a university SOC was rapidly adopted by analysts — when outputs matched existing verification workflows and could be tailored, users reused the AI's analysis in over 90% of cases and prolific users reported roughly 30–50% faster ticket handling.

Main Finding

When an agentic AI companion for Security Operations Center (SOC) triage is designed "in the trenches" with users, presents its findings in verifiable, analyst-familiar formats, and is made adaptable by analysts, it is rapidly adopted and materially reduces analyst effort. In the study site, analysts reused the companion’s outputs in ticket closing reports in over 90% of cases, and the most active users reported estimated per-ticket time savings of roughly 30–50%.

Key Points

• Intervention and setting

  • 14-month embedded field study in a university SOC. Two PhD researchers worked as tier‑1 analysts (participant observation). The companion was used by other experienced analysts voluntarily during the final 4 months.
  • Use was real-world and observational (not a controlled experiment); IRB-approved and voluntary.

• System design

  • ReAct-style LLM agent (LangChain) running on locally hosted models (Ollama) on an NVIDIA DGX Spark.
  • Tool layer exposes read-only interfaces to ticketing, SIEM/log search, DHCP/MAC/device registries, identity portal, reputation/threat intel, and utilities. All tool calls are read-only and schema-validated.
  • Outputs include structured SOC-formatted reports and captured visual evidence (headless browser screenshots) of tool results.
  • Analysts can adapt the agent via editable, versioned system prompts that encode workflow order, abandonment rules, and report templates.

• Adoption and behavior

  • Analysts reused the companion’s outputs in the closing report in >90% of cases where it was used.
  • Analysts naturally personalized system prompts and kept several named prompt versions, switching per investigation.
  • Making evidence presentation verifiable (same format as manual process, raw tool responses, screenshots) was critical to trust and adoption—accuracy alone was not sufficient.
  • Analysts could interrupt, ask follow-ups, or branch runs; the companion did not make state changes to SOC systems.

• Productivity and perceived effects

  • Estimated time savings for the most prolific Companion users: ~30–50% per-ticket processing time (compared to self-reported manual time).
  • Analysts reported the time spent handling tickets with the companion felt more productive.
  • The more analysts shaped the companion (prompt customization), the more they trusted and used it.

• Refinements and practical design lessons

  • Capturing screenshots of evidence pages and aligning report layout to analysts’ expectations materially increased adoption.
  • A shared base prompt plus personal versions provided a balance of defaults and individual adaptability.

• Limitations

  • Single-site study (university SOC) with many student analysts; findings may not generalize to all enterprise SOCs.
  • Observational deployment (no randomized control); time-savings are estimated and self-reported for manual baseline.
  • Locally hosted models may limit model capabilities vs. latest cloud models, though that trade-off was deliberate for data governance.

Data & Methods

• Study design

  • Longitudinal, embedded participant-observation over ~14 months inside one SOC.
  • Two researchers functioned as tier‑1 analysts to learn workflows and drive iterative design.
  • Final 4-month voluntary deployment to several experienced SOC analysts.

• Data sources

  • Field notes and reflective memos documenting workflow, delays, tool use, and informal feedback.
  • Companion logs: prompts, AI responses, tool calls and outputs, versioned system prompts, timestamps, and transcripts of user-agent interactions.
  • Ticketing system records: final analyst reports and metadata showing whether companion output was reused.

• Agent implementation specifics

  • ReAct agent loop alternating reasoning and tool calls, terminating when it can produce a structured report.
  • Tool calls executed on server; headless browser snapshots captured for each lookup.
  • Local models (run via Ollama) selectable per run; tool APIs are read-only; no external leakage of internal identifiers.

• Analysis

  • Mixed-methods: qualitative analysis of field notes and transcripts; quantitative analysis of logs and ticket reuse rates, plus time-savings estimates based on timestamps and self-reports.
  • Research questions addressed: (1) Do analysts use the companion’s outputs? (2) Do analysts try to modify its behavior? (3) What improvements (or not) are observed?

Implications for AI Economics

• Adoption drivers and product design

  • Verifiability > raw model accuracy for practitioner adoption in high‑stakes, accountable tasks. Economic models of AI adoption should weight evidence-format and verification costs heavily.
  • Customizability (user-editable system prompts, per-user workflows) materially increases uptake—markets will value platforms that enable easy local adaptation over rigid one‑size‑fits‑all offerings.

• Labor impact and complementarities

  • Agentic companions can automate repetitive evidence-assembly tasks, reducing time spent on low-skill repetitive steps while leaving judgment to humans—this implies partial task displacement rather than full job loss, increasing demand for higher‑skill oversight and prompt/system-engineering capabilities.
  • Productivity gains (30–50% per-ticket for active users) translate into labor-cost savings or capacity increases; firms may reallocate analyst time to higher-value incident response, threat hunting, or coverage expansion.

• Cost structure and deployment choices

  • Effective deployment requires engineering and compute resources (DGX-class hardware in study), integration effort with internal toolchains, and ongoing maintenance to keep tool interfaces and prompts aligned to changing workflows.
  • Data governance concerns push organizations toward local hosting despite potentially higher model/infra costs—this affects the TCO and the competitive landscape (in-house solutions vs. vendor cloud services).

• Market and organizational implications

  • Vendors: opportunity to offer adaptable, transparent agent frameworks (versioned prompts, evidence capture, read-only tool APIs). Selling “companions” as modifiable platforms rather than closed features may accelerate enterprise uptake.
  • New services and labor markets: prompt engineering, internal agent orchestration, UI for evidence-verification, and SOC‑specific agent integration become valuable skills/services.
  • Heterogeneous adoption: smaller organizations with weak internal engineering may prefer vendor-managed offerings; larger orgs or those with strict data governance will invest in local, customizable systems.

• Policy, liability, and incentives

  • Liability remains with human analysts; economic incentives will favor designs that minimize verification costs and make audit trails explicit (evidence screenshots, raw tool outputs).
  • Regulators or insurers may incentivize verifiable evidence formats and read-only operations for automated assistants in security operations.

• For economic modeling and empirical work

  • Future models should incorporate: verification cost (time analysts spend validating AI output), adaptability benefit (value from personalization), compute & integration fixed costs, and heterogeneity in analyst workflows.
  • Empirical research should measure realized labor reallocation (what analysts do with freed time), long-run retention/turnover effects, and cross-site generalizability to enterprise SOCs.

Summary takeaway Design agentic AI companions as adaptable, verifiable partners embedded in existing workflows. Economic value accrues primarily from reducing repetitive evidence-assembly time while preserving human judgment; adoption depends critically on how easily outputs can be verified and personalized to local practice.