0 cumulative citations
View corpus contextA self-evolving 'harness' that supplies system-specific expertise sharply improves LLM-based root-cause analysis—lifting top-1 accuracy to 59% and more than quadrupling performance over prior specialized agents—though gains rely on high-quality telemetry, integration, and the chosen model backbone.
From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis
OpsHarness is a self-evolving external harness for general LLM agents that mines and reuses system-specific diagnostic experience to markedly improve automated root-cause analysis accuracy on public benchmarks and in a production deployment.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Automated root cause analysis (RCA) with large language models (LLMs) has drawn growing attention. Today, SREs typically automate RCA with LLMs in one of two ways: directly using a general-purpose agent (e.g., Codex or Claude Code) for diagnosis, or building a specialized RCA agent from scratch. As mainstream general agents grow more capable and iterate quickly, our quantitative study finds that the former now often surpasses the latter. Its accuracy, however, still falls short of production needs, and this gap stems mainly from the external adaptation layer outside the agent's general capabilities, namely the harness. We therefore argue that LLM-based RCA should focus on this external harness, reusing the strong general capabilities of a modern agent rather than rebuilding an agent from scratch. A key capability of such a harness is to self-evolve, accumulating system-specific experience from past diagnoses so that it gets better the more it is used. We introduce OpsHarness, a self-evolving RCA harness that turns diagnosis experience into reusable expertise. Its data plane combines layered operational knowledge with an idea-card tool library, while its control plane coordinates setup, diagnosis, evolution, and verification. During evolution, OpsHarness contrasts successful and failed trajectories, converts their evidence into atomic proposals, and admits updates only through a dual-gate verification process designed to prevent overfitting and regression. Across two public benchmarks and an industrial deployment, OpsHarness achieves 59.0\% top-1 accuracy, improving over a bare general agent by 63.4\% and over baseline RCA agents by 4.02$\times$.
Summary
Main Finding
OpsHarness — a self-evolving external harness layered on top of modern general-purpose LLM agents — substantially improves automated root-cause analysis (RCA) accuracy versus (a) a bare general agent and (b) specialized RCA agents built from scratch. Averaged across four LLM backbones and two public benchmarks plus an industrial deployment, OpsHarness reaches 59.0% top-1 accuracy, a 63.4% relative gain over a bare general agent and a 4.02× improvement over baseline specialized RCA agents.
Key Points
-
Why harness, not a new agent:
- Modern general-purpose agent frameworks (e.g., Codex / Claude Code) already supply mature tool use, sandboxing, long-horizon planning and context management. Rebuilding these capabilities in a specialized RCA agent often underperforms.
- The main remaining gap is system-specific diagnostic expertise and repeated-incidents experience — i.e., what sits outside the agent: the external harness.
-
OpsHarness design (high level):
- External harness compatible with general agents (reuses the agent’s general capabilities).
- Two logical planes:
- Data plane h = (K, T):
- K: layered operational knowledge store with four tiers:
- K0: general RCA background and guidelines (shipped).
- K1: per-system profile (schema, components, telemetry paths) produced at setup.
- K2: mined workflow skeletons (preferred diagnostic paths).
- K3: mined atomic diagnostic operations and rules (positive/negative patterns).
- T: idea-card tool library with progressive disclosure (an organized set of diagnostic tools/operations).
- K: layered operational knowledge store with four tiers:
- Control plane: lifecycle workflows — setup, diagnose, evolve, verify — plus observability that records every run (session ID, trajectory JSON).
- Data plane h = (K, T):
- Self-evolution loop:
- Evolve: mines trajectories (both successful and failed) into atomic evolution proposals (extracts reusable practices, caveats, propagation chains).
- Verify: dual-gate sandbox + held-out testing to prevent overfitting and regression; only proposals that pass are promoted into the harness.
-
Empirical evidence:
- Benchmarks: OpenRCA and RCAEval; industrial deployment and case studies (Company A).
- Models/backbones tested: GPT-5.5 (Codex), Claude Sonnet 4.6 (Claude Code), GLM-5.2, DeepSeek-V4.
- Baselines: bare general agents vs specialized RCA-Agent and mABC.
- Representative numeric comparisons (from paper):
- General agent (GPT-5.5 Codex): OpenRCA 44.9%, RCAEval 57.4% (overall 51.2%) vs RCA-Agent 31.8%, 1.9% (overall 16.8%).
- Overall across backbones: general agents average 36.1% top-1, RCA-Agent 17.9%, mABC 5.6%.
- OpsHarness averaged 59.0% top-1 across four backbones — large gains vs both bare agents and specialized agents.
- Practical case: shows how mining a corrected diagnosis (e.g., distinguishing misleading noisy DB session counts vs true CPU saturation) guides future incidents and reduces MTTR.
-
Operational characteristics:
- Overhead comparable to a bare general agent.
- Continuous improvement: performance increases as the harness accumulates experience from repeated diagnoses.
- Safety: dual-gate verification aims to limit overfitting and regressions from automatic evolution.
Data & Methods
- Datasets and deployments:
- Two public RCA benchmarks: OpenRCA and RCAEval.
- Industrial production case studies and deployment data from a large cloud provider (Company A).
- Models and baselines:
- Four LLM backbones evaluated via general-purpose agent frameworks (Codex, Claude Code).
- Compared to specialized RCA agents: RCA-Agent and mABC.
- Measurements:
- Primary metric: top-1 root-cause localization accuracy.
- Additional evaluation of evolution effectiveness (improvement over time), per-component ablations (to measure contribution of K0–K3, tool library, verification).
- Methodology highlights:
- Each run recorded as an observability trajectory (session ID, structured JSON) for later mining.
- Evolution mines both successful and failed trajectories into atomic proposals (operations, rules, workflow skeletons).
- Verification uses a sandbox run (source system) and held-out tests to promote only non-regressing changes.
- Experiments measured average accuracy across backbones and systems, plus industrial MTTR/case comparisons.
Implications for AI Economics
- Higher marginal return from harness engineering than re-training models:
- As base LLM agents rapidly improve, incremental RCA performance is driven more by task-specific harnessing (knowledge, tooling, workflows) than by model improvements. Investing in external harnesses can yield outsized, cost-effective gains relative to bespoke agent development or new model training.
- Lower engineering cost & faster deployment:
- Reusing general agents plus a harness reduces duplicate engineering effort (no re-implementing robust tool-use, sandboxing, planning). Faster time-to-value for enterprises building RCA automation.
- Data and feedback as valuable assets:
- The harness turns incident trajectories and engineer feedback into reusable, monetizable assets (K2/K3). Firms that capture and verify diagnostic trajectories effectively build a proprietary dataset that enhances automation performance over time — a competitive data moat.
- Business model and productization opportunities:
- Market for harness components (idea-card tool libraries, verification sandboxes, per-system profile adapters, evolution modules). Services: managed harness development, cross-customer pattern extraction (with privacy), verification-as-a-service.
- Labor and skill-shift:
- Augmentation, not wholesale replacement: OpsHarness reduces routine MTTR and wrong-turns, but SREs remain needed for verification, complex remediation, and handling novel incidents. Demand shifts toward skills in harness engineering, verification, feedback curation, and governance.
- Risk management and regulatory considerations:
- Automated self-evolution introduces risks (overfitting to idiosyncratic incidents, silent regressions). OpsHarness’s dual-gate verification addresses this, but firms will need auditability, human-in-the-loop sign-off policies, and liability frameworks when promoting automated remedies into production.
- Platform lock-in and interoperability:
- Harnesses are system-specific but designed to sit atop evolving general agents. This creates incentives to standardize harness interfaces (to switch backbones without redoing harness logic) and to build marketplaces for verified idea-cards/rules. Conversely, proprietary harness data can increase switching costs.
- Broader productivity and economic impact:
- Continuous learning harnesses can reduce MTTR and operational costs across large fleets, improving service reliability and reducing downtime costs. The economic gains scale with incident recurrence frequency and the breadth of telemetry integrated.
Caveats and limits: - OpsHarness improves performance substantially but is not perfect (59% top-1 averaged). Performance depends on availability of rich telemetry, incident feedback, and repeated incident patterns. - Evolution depends on high-quality labels/feedback; noisy or absent feedback constrains learning. - Generalization across very different systems still requires per-system K1 profiling and mining; cross-system transferables may be limited without careful abstraction.
Summary takeaway: In modern LLM-enabled automation, the external harness — especially one that self-evolves by mining and verifying operational experience — is the high-leverage place to invest. For firms and product teams, building verifiable harness infrastructure and treating incident trajectories as strategic data assets will likely yield larger, faster economic returns than focusing primarily on new model development for RCA.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports controlled comparative experiments on two public benchmarks and an industrial deployment, showing substantial accuracy gains versus both bare general agents and prior specialized RCA agents; however, the evidence is not a randomized causal design, sample sizes and statistical tests are not reported in the supplied text, and the industrial evidence appears limited to a single company, reducing confidence in external validity.
Methods Rigormedium — The authors compare OpsHarness against multiple baselines across four backbone models and both public benchmarks plus an industrial deployment, and they describe an evolution/verification procedure and ablation-style component claims; however, the provided excerpt lacks detail on dataset sizes, held-out splits, statistical significance, hyperparameter tuning, and robustness checks, making it difficult to fully judge rigor.
SampleEvaluation uses two public benchmarks for RCA (OpenRCA and RCAEval) and an industrial production deployment at a large cloud service provider (Company A); experiments are run across four model backbones (e.g., GPT-5.5/Codex, Claude Code, GLM-5.2, DeepSeek-V4) and compare a bare general-purpose agent, specialized RCA agents (RCA-Agent, mABC), and OpsHarness-enhanced agents. Exact counts of queries/episodes, split sizes, and number of industrial incidents are not provided in the supplied text.
Themesproductivity human_ai_collab
GeneralizabilityBenchmarks (OpenRCA, RCAEval) may not cover the full diversity of real-world systems and failure modes., Industrial deployment evidence appears to come from a single company, limiting cross-organization generalizability., Performance depends on the capabilities of the underlying general agent/backbone; results may change with different or weaker models., Requires rich telemetry, tool integrations, and operational metadata—environments lacking these will see smaller gains., Potential for overfitting to recurring incident patterns in a single system despite verification safeguards.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Across four backbone models and two public RCA benchmarks, general-purpose agents achieved higher overall top-1 accuracy than specialized RCA agents on every tested backbone. Decision Quality | positive | Top-1 root cause identification accuracy |
Reading fidelity
high
Study strength
high
|
36.1% overall accuracy for general agents versus 17.9% for RCA-Agent and 5.6% for mABC
|
| On the OpenRCA benchmark, bare Codex using GPT-5.5 achieved 44.9% top-1 accuracy, compared with 31.8% for RCA-Agent using the same backbone. Decision Quality | positive | OpenRCA top-1 accuracy |
Reading fidelity
high
Study strength
high
|
44.9% versus 31.8%
|
| The specialized RCA-Agent generalized poorly from OpenRCA to RCAEval, achieving only 5.1% average accuracy on RCAEval, while the general agent achieved 38.0%. Decision Quality | negative | RCAEval top-1 accuracy and cross-benchmark generalization |
Reading fidelity
high
Study strength
medium
|
5.1% for specialized RCA-Agent versus 38.0% for the general agent
|
| The strongest bare general-agent configuration tested, Codex with GPT-5.5, achieved an average top-1 accuracy of 51.2% across the test datasets. Decision Quality | positive | Average top-1 root cause identification accuracy |
Reading fidelity
high
Study strength
high
|
51.2% average accuracy
|
| OpsHarness achieved 59.0% top-1 accuracy averaged over the four tested backbones and improved over a bare general agent by 63.4% relatively. Decision Quality | positive | Average top-1 RCA accuracy |
Reading fidelity
high
Study strength
medium
|
59.0% top-1 accuracy; 63.4% relative gain
|
| OpsHarness achieved 4.02 times the performance of the baseline specialized RCA agents. Decision Quality | positive | Top-1 RCA accuracy relative to specialized-agent baselines |
Reading fidelity
high
Study strength
medium
|
4.02× improvement
|
| The paper reports that OpsHarness continuously improves during the diagnosis process through self-evolution, with each design component contributing to the overall performance gains. Decision Quality | positive | RCA diagnosis performance over successive self-evolution stages |
Reading fidelity
high
Study strength
low
|
not reported
|
| In a Company A production incident, an initial database-connection diagnosis was incorrect; the actual root cause was CPU saturation on an associated container. Decision Quality | negative | Root cause localization correctness |
Reading fidelity
high
Study strength
medium
|
n=1
|
| In the production case study, reusing lessons from the first incident enabled the SRE to localize the faulty container more directly in a subsequent incident, with fewer wrong turns and lower mean time to resolution. Task Completion Time | positive | Diagnostic efficiency and time to resolve an incident |
Reading fidelity
high
Study strength
low
|
n=2
|
| Attaching the correct troubleshooting guide reduced the mean time to mitigate a severity-2 incident from about 18 hours to about 13 hours. Task Completion Time | positive | Mean time to mitigate a severity-2 incident |
Reading fidelity
high
Study strength
medium
|
about 18 to 13 hours
|