A governed AI workflow slashes crystal-structure analysis time at the Spallation Neutron Source by about fivefold while preserving verification and provenance, delivering publication-ready outputs; the agent runs on two LLM backends and enforces allowlisted tools and fail-closed gates to maintain traceability.

Main Finding

NeuDiff Agent is a governed, tool-using AI workflow prototype that accelerates end-to-end TOPAZ single-crystal neutron diffraction analysis (instrument data → validated CIF) while preserving facility governance. In a controlled reference-case benchmark, NeuDiff Agent reduced wall time from 435 minutes (manual) to 86.5 ± 4.7 and 94.4 ± 3.5 minutes (two LLM backends; 4.6–5.0× faster) and produced publication-ready CIFs with no checkCIF level A or B alerts. The system couples constrained tool invocation, fail-closed verification gates, and a provenance-first audit/replay record so outputs are auditable and replayable.

Key Points

• Purpose and positioning

  • Targets instrument-to-publication latency at large-scale facilities (TOPAZ, SNS) where iterative reduction, integration, refinement and validation cause bottlenecks.
  • Emphasizes governance: allowlisted tools, deterministic verification gates that fail-closed, and complete provenance capture to preserve traceability and user accountability.

• Architecture and components

  • LLM Core wrapped by LangGraph state machine: structured prompts (system role, neutron knowledge schema, chat history, tool logs), emits either text or structured function calls.
  • Constrained tool interface: audited function wrappers for Mantid/TOPAZ ReductionGUI, SHELXL, IUCr checkCIF, visualization (VESTA), etc. Functions are validated, idempotent where possible, and write artifacts to experiment workspaces.
  • Model Context Manager: four memory types — system prompt, neutron knowledge schema, chat-history, and log-message memory (tool-call records).
  • RAG (retrieval-augmented generation) from a local, versioned document store (instrument docs, parameter refs, checkCIF guidance); retrieval releases are immutable and logged.
  • Browser UI built on NOVA/Trame: free-text prompts, structured controls, explicit user authorization before actions that change results.

• Governance mechanisms

  • Verification gates (fail-closed) at workflow boundaries: schema/hard-bounds, cross-input consistency, tool-verified execution checks, and publication validation (checkCIF).
  • Intervention events recorded when gates fail; workflow halts and requests user-authorized corrective actions.
  • Per-run provenance bundles capture prompts, configs, tool-call args & outputs, exit codes, warnings, timestamps, state transitions — enabling inspection and controlled replay.

• Evaluation protocol and outcomes

  • Reference dataset: TOPAZ scolecite dataset, using original X-ray CIF as starting model.
  • Controlled evaluation: fixed prompt list, repeated end-to-end runs, two LLM backends, five runs per backend to estimate variability.
  • Metrics extracted from logs: partitioned human vs machine time, wall-time reduction, intervention frequency, failure modes, recovery behavior.
  • Result: 4.6–5.0× wall-time speedup vs a single expert manual run while meeting publication validation (no level A/B checkCIF alerts).

• Limitations mentioned

  • Prototype tested on a representative reference case and some additional materials; broader generality requires more datasets and deployment testing.
  • Designed to concentrate human judgment at explicit checkpoints rather than fully automate scientific decisions.

Data & Methods

• Data

  • Primary benchmark dataset: TOPAZ single-crystal neutron diffraction scolecite dataset; starting structure borrowed from X-ray CIF for initialization.
  • Facility documents and validation references used by RAG: TOPAZ ReductionGUI docs, instrument geometry, operating bounds, IUCr checkCIF guidance. Documents are versioned and locally indexed.
  • Execution logs and artifacts: reduction configurations (.config), Mantid outputs, HKLF2 reflection files, SHELXL outputs, final CIFs, checkCIF reports.

• Methods / System implementation

  • Workflow formalization: explicit LangGraph state machine with typed workflow variables covering reduction → integration → refinement → validation.
  • LLM interaction model: structured prompts with constrained allowed actions; function-call pattern where tool calls are executed by Python wrappers and logged.
  • Verification checks (examples)
    • Hard bounds: unit-cell plausibility, wavelength/d-spacing and resolution sanity, required file formats.
    • Cross-input checks: space group vs unit cell, UB matrix vs orientation metadata, run-list vs calibrations.
    • Tool-verified checks: Mantid log signatures, SHELXL convergence, displacement parameter plausibility.
    • Publication checks: CIF metadata completeness and checkCIF alerts (A/B must be cleared).
  • User interaction model: UI requires explicit authorization before executing steps that change analyses; ambiguous requests are disambiguated by LLM mapping to high-level actions.
  • Provenance and replay: per-run audit directories capturing everything required to reconstruct analysis path: prompts, config files, tool args/outputs/statuses, gate outcomes, change logs and rationale.
  • Evaluation protocol: fixed prompt list converted from an unaided manual run; repeated runs to measure variability; partitioning of wall time into human and machine components; two LLM backends tested with pinned tool and retrieval config.

Implications for AI Economics

• Throughput and cost-efficiency

  • Direct productivity gains: 4.6–5.0× reduction in wall time for the representative case implies substantial increases in effective instrument throughput and analyst productivity, reducing per-result labor hours and potentially lowering marginal cost per validated structure.
  • Better utilization of expensive facility time: faster instrument-to-publication turnaround can increase scientific output per allocated beamtime and may permit higher facility utilization/value capture.

• Labor impacts and task reallocation

  • Shift from routine bookkeeping toward checkpointed expert decisions: NeuDiff concentrates human intervention at explicit gates, likely reducing time spent on repetitive configuration and rework while increasing the relative importance of domain judgment at critical points.
  • Complementarity rather than full substitution: the agent automates many steps but requires expert authorization at governance checkpoints — suggesting role changes (more oversight, method validation, and anomaly resolution) and upskilling rather than wholesale displacement.

• Risk management, governance value, and adoption incentives

  • Provenance-first design dramatically lowers audit and compliance costs: deterministic logs, repeatability, and recorded rationales reduce organizational risk and liability concerns associated with black-box automation.
  • Fail-closed gates reduce the probability of silent failures propagating downstream, which decreases costly rework and reputational risk from publishing flawed results — improving the ROI of automation in regulated or high-stakes scientific environments.
  • These governance properties increase organizational willingness to deploy agentic systems in production, making adoption economically more feasible for large facilities.

• Productization and scaling economics

  • Marginal cost structure: once tool wrappers, retrieval corpora, and governance rules are in place, additional runs scale with compute and modest human oversight; incremental cost per additional validated CIF should decline.
  • Upfront investment: building allowlisted integrations, curated knowledge bases, interface/authorization workflows, and audit/replay tooling entails fixed development and validation costs that justify deployment only if throughput gains are sustained across many runs.
  • Standardization benefits: a validated, provenance-first workflow can be commodified as a product/service across comparable facilities (neutron/X-ray scattering) producing economies of scale and potential shared infrastructure.

• Externalities and induced demand

  • Faster turnaround can increase demand for follow-up experiments, re-analyses, and exploratory runs (induced demand), which may raise overall facility usage and revenue but also require capacity planning.
  • Lower friction may shift incentives toward more ambitious or iterative scientific programs, changing the portfolio of experiments performed.

• Policy, compliance, and market effects

  • Demonstrated governance mechanisms (allowlist, fail-closed gates, immutable retrieval releases, auditable logs) create a template for regulatory compliance and standards development in AI-assisted scientific workflows.
  • Facilities that can demonstrate provenance and gated verification will have a competitive advantage in attracting users with publication or compliance constraints.

• Measurement and valuation recommendations

  • Facilities should track both machine and human time separately, along with intervention frequency, gate-failure causes, and downstream rework rates to compute true ROI.
  • Consider valuing the system not only by wall-time saved but by reductions in rework, improved reproducibility, and lowered publication risk.

Overall, NeuDiff Agent illustrates how governance-first agent design shifts the economics of high-stakes scientific workflows: it reduces per-result labor and error risk, concentrates human expertise where it matters, and creates a durable audit trail that lowers organizational barriers to adopting agentic automation. The approach requires non-trivial upfront investment in integration and curation but yields recurring operational savings and risk-reduction value that can justify deployment at scale.

Limitations and next steps (economic lens) - Prototype tested on a constrained set of datasets and a fixed prompt protocol; broader validation is needed to estimate realized gains across diverse sample types and instrument modes. - Cost-benefit analyses should include development, validation, continued curation, and governance overheads versus recurring time-savings and induced revenue from increased throughput. - Future work could quantify labor reallocation effects, training needs, and market implications for service providers that package provenance-first scientific agents.