An affordable, vendor-agnostic blueprint for self-healing data and AI pipelines: the authors show how commodity open-source telemetry, lineage, policy gates, and LLM-based diagnosis can be composed into a guarded, agentic remediation loop — but the design is architectural and remains unvalidated in production.

Main Finding

Modern self-healing capabilities for data, ML, and software delivery pipelines can be assembled affordably and vendor‑agnostically from mature open-source components and inexpensive LLMs. The main gap is architectural: existing vendor solutions bundle the needed pieces into closed ecosystems that deliver high autonomy at the cost of lock‑in and high price. The paper proposes a modular reference architecture (Agentic Recovery and Incident Response) that combines telemetry, metadata/lineage, deterministic policy checks, LLM‑assisted diagnosis, approval workflows, guarded remediation, verification, and learning — designed so small teams can incrementally deploy self‑healing without committing to a single vendor.

Key Points

• Why pipelines fail: failures cluster into six classes — data quality, schema change, upstream source change, infrastructure faults, orchestration failures, and model workflow issues. Root causes often lie many hops upstream, so lineage/metadata is central to diagnosis.
• The self‑healing loop comprises eight stages: detect, triage, diagnose, plan, approve, remediate, verify, learn. Automation should route well‑known failures to deterministic rules and reserve LLM agents for contextual diagnosis.
• Market survey: eight commercial products were qualitatively compared (Databricks Genie ZeroOps, Acceldata ADM, Dynatrace Davis AI + Workflows, Datadog Bits AI + Workflow Automation + Data Observability, Monte Carlo, IBM Databand, PagerDuty SRE Agent + Runbook Automation, ServiceNow Predictive AIOps + Autonomous Workforce + AI Control Tower).
  • Evaluation dimensions: primary scope, detection & RCA, remediation autonomy, vendor coupling, indicative cost.
  • Three structural findings:
  • Greater autonomy correlates with deeper vendor lock‑in (platforms that close the loop tend to own catalog/orchestration/CMDB).
  • No single product fully covers data plane, application/infrastructure plane, and model quality plane.
  • All platforms share convergent ingredients: telemetry, metadata/topology, inference layer (statistical/causal/LLM), policy/approval gates, and an action runtime — each with open‑source counterparts.
• Proposed reference architecture (Agentic Recovery and Incident Response):
  • Design principles: vendor agnosticism via open interfaces (OpenTelemetry, Prometheus, OpenLineage), model swap‑ability, minimal privileges for agents, human‑centered approval gates, auditable incident store, iterative/lowest‑privilege implementation path.
  • Core components: telemetry collectors, lineage/catalog, incident/event store, deterministic policy engine (for known fixes), LLM agent(s) with tool access (queries over metrics/logs/lineage/runbooks), approval/workflow service, guarded remediation runtime (parameterized, audited), verification checks, incident learning store for retrieval‑grounded RCA.
  • Implementation guidance: modular, pick small viable toolset; recommended patterns include retrieval augmentation for grounding LLMs, deterministic routing of routine failures to scripted remediation, and human‑in‑loop for higher‑risk actions.
• Limitations and risks highlighted: security and privilege management, hallucination and incorrect remediation by LLMs, incomplete telemetry or lineage limits effectiveness, governance/ auditing requirements, and that the architecture is prescriptive (design synthesis) rather than empirically validated at scale.

Data & Methods

• Method: qualitative systems and literature synthesis rather than new empirical experiments.
  • Surveyed vendor documentation and product descriptions (mid‑2026 snapshot) to compare eight commercial platforms.
  • Constructed a simple ordinal rubric (none/partial/good/strong) across capability dimensions: monitoring/detection, data‑quality awareness, root‑cause analysis, automated remediation, cross‑domain support, governance/auditability, open‑source friendliness, low cost, and vendor‑agnosticism.
  • Positioned platforms on two axes (remediation autonomy vs vendor coupling) to expose trade‑offs.
• Design method for architecture: distillation of recurring patterns across the surveyed platforms and recent AIOps / LLM agent literature; proposed concrete, illustrative open standards and OSS components (e.g., OpenTelemetry, Prometheus exposition, OpenLineage; agent frameworks and retrieval patterns referenced).
• Validation status: conceptual and prescriptive; no production benchmarking, cost modeling, or randomized evaluation reported in the paper. The architecture is accompanied by buy‑vs‑build guidance and discussion of practical tradeoffs, but empirical ROI and safety testing are left as next work.

Implications for AI Economics

• Cost structure and vendor dynamics
  • Lowers entry barrier: modular open‑source stacks + inexpensive LLM inference can enable small teams (2–4 engineers) to implement meaningful self‑healing, reducing reliance on high‑cost vendor suites.
  • Upward pressure on vendor pricing: if modular architectures become common, demand for vertically integrated “autonomous” platforms may soften, pushing vendors to compete more on integration, governance, and value‑add services.
  • New operational costs remain: engineering time to integrate telemetry/lineage, run LLMs, secure agents, and maintain policy/rules; LLM inference and storage costs are nonzero though likely lower than enterprise SaaS per‑entity billing in many estates.
• Labor and task reallocation
  • Routine triage and repetitive remediation tasks can be automated, reducing time engineers spend on incident cleanup and shifting labor toward building resilience, observability, and higher‑value model/data work.
  • Demand shifts: more engineering effort will be needed for metadata capture, lineage instrumentation, policy engineering, and governance — specialized roles (incident policy engineers, SREs skilled in agent safety) may grow.
• Investment and procurement decisions
  • Firms will face explicit buy vs build tradeoffs: turnkey vendor bundles offer faster time‑to‑autonomy at higher recurring cost and lock‑in; the proposed architecture offers lower recurring licensing costs and greater flexibility but requires upfront integration and operational skill.
  • ROI hinges on estate heterogeneity and failure costs: heterogeneous estates and medium‑sized teams likely benefit most from vendor‑agnostic stacks; greenfield or platform‑standardized estates may still prefer integrated vendor offerings.
• Market and competition effects
  • Increased interoperability (via OpenTelemetry, OpenLineage) could enable specialist vendors and open‑source projects to compete on niche capabilities (model quality monitoring, lineage‑aware RCA, secure remediation runtimes).
  • Platform incumbents may counteract by offering better governance, audited automation primitives, and developer ergonomics to retain lock‑in value.
• Externalities, governance, and regulation
  • Autonomous remediation raises accountability and auditability needs; the architecture’s emphasis on approval gates, minimal privilege, and incident stores responds to these economic and regulatory risks.
  • Insufficient governance or poor grounding of LLMs could produce costly misremediations; firms must internalize these risk management costs.
• Research and measurement needs (economic research agenda)
  • Empirical evaluation of ROI: controlled studies comparing vendor bundles vs open modular stacks on TCO and incident MTTR.
  • Labor effects: studies on task reallocation, productivity gains, and wage impacts for SRE/ML engineers.
  • Incentives for metadata capture: quantifying returns to investment in lineage/observability for diagnoseability and automation.
  • Safety/economic tradeoffs: models for optimal human‑in‑loop policy thresholds balancing automation benefits against risk costs.

Summary takeaway for AI economists: the paper argues that the components for affordable, practical self‑healing are now commoditized; the remaining barrier is architecture and integration. This creates a plausible path to reduce recurrent vendorized spending and to reallocate labor to higher‑value activities—but realizing those gains requires investment in metadata, governance, and safe agent design.