An AI 'Test Automation Copilot' sped up regression test-script production at Hacon and reused 30–50% of code, boosting throughput; however, maintainability and domain-correctness still require human review and governance.

Human-AI Collaboration for Scaling Agile Regression Testing: An Agentic-AI Teammate from Manual to Automated Testing

Moustapha El Outmani, M. Shenoy, A. Hatahet, Andreas Rausch, T. Kniep, T. Raddatz, Benjamin King · Fetched April 17, 2026

semantic_scholar descriptive low evidence 7/10 relevance Full text usable extracted full text Source PDF

An agentic Test Automation Copilot integrated into Hacon's CI pipelines accelerated test-script authoring and increased throughput with 30–50% code reuse, but human review remains necessary for correctness and maintainability.

Automated regression testing is essential for maintaining rapid, high-quality delivery in Agile and Scrum organizations. Many teams, including Hacon (a Siemens company), face a persistent gap: validated test specifications accumulate faster than they are automated, limiting regression coverage and increasing manual work. This paper reports an exploratory industrial case study of the Hacon Test Automation Copilot, an agentic AI system that generates system-level regression test scripts from validated specifications using retrieval-augmented generation and a multi-agent workflow. Integrated with Hacon's CI pipelines, the Copilot operates asynchronously as a"silent AI teammate", producing candidate scripts for human review. Mixed-method evaluation shows the AI accelerates script authoring and increases throughput, with 30-50% code reuse. However, human review remains necessary for maintainability and correct domain interpretation. Clear specifications, explicit governance, and ongoing human-AI collaboration are critical. We conclude with lessons for scaling regression automation and enabling effective human-AI teaming in Agile settings.

Summary

Main Finding

An agentic AI “Test Automation Copilot” integrated into an industrial Agile testing workflow materially accelerates the conversion of validated manual test specifications into executable regression scripts (with 30–50% of generated code reused), but most generated scripts still require moderate-to-major human edits for maintainability, correct domain interpretation, and production readiness. Clear specifications, explicit governance, and ongoing human–AI collaboration are therefore necessary to realize sustainable automation gains.

Key Points

System design
- Agentic, asynchronous “silent AI teammate” that produces candidate system-level test scripts ahead of sprints.
- Uses retrieval-augmented generation (RAG) over historical spec→script pairs and a bounded multi-agent loop (Generator → Execute in Jenkins → Evaluator → Reporter).
- Outputs include generated script, Jenkins execution logs, MLflow execution traces, and structured summaries for human review.
- Bounded autonomy: AI cannot add scripts to the regression suite without human approval; all artifacts are recorded for traceability.
Empirical outcomes
- Code reuse: 30–50% of AI-generated code was retained unchanged on average.
- Edit distribution (n = 49 generated scripts analyzed):
  - 15 fully rewritten
  - 20 required major edits
  - 13 required medium edits
  - 1 required minor edits
- Common quality issues affecting maintainability:
  - Hardcoded test data: 31/49
  - Redundant imports: 29/49
  - Unused objects: 23/49
  - Missing/incorrect validations: >30 cases
- Human reviewers often replaced AI blocks with familiar, trusted snippets even when AI logic was correct—preference for maintainability and established conventions.
Process & governance lessons
- Specification clarity is foundational; AI performance degrades with ambiguous or tacit-knowledge-heavy specs.
- Human oversight remains indispensable for semantic correctness, domain interpretation, and long-term maintainability.
- Co-adaptation (improving prompts, feedback loops, and spec-writing practices) is required to scale benefits.
Limitations
- Single-company case study, small team (five engineers), short evaluation window (4 weeks) — limits generalizability.
- Practitioner bias and selection of test cases may affect results.

Data & Methods

Context: Industrial case study at Hacon (a Siemens company) within a mature Agile/CI environment.
Duration & participants: 4-week deployment, 5 test engineers.
Dataset: 61 validated test specifications (Xray format) across 6 functional areas; complexity ranged 2–18 steps per case; story point estimates 3–8; pre-rated input clarity A–D.
Procedure:
- Copilot generated scripts for all 61 specs.
- Engineers reviewed and refactored generated scripts at sprint start.
- Structured survey per script (block-level correctness, quality, readability, usability, test smells; 1–5 Likert + open text).
- Detailed artifact comparison on a representative subset (~20 cases) by two expert engineers using version-control diffs to assess semantic equivalence, maintainability, and recurring issues.
Instrumentation: Jenkins for CI execution, MLflow for traceability, version control for artifact comparisons.
Key quantitative measures: percent code retained, counts of common defects/anti-patterns in generated scripts, edit severity distribution.

Implications for AI Economics

Productivity vs. augmentation
- Partial automation: 30–50% code reuse indicates the AI provides a meaningful head start but not full substitution for skilled engineers. Economic gains are thus in augmented throughput rather than labor replacement.
- Time savings are task- and context-dependent; many cases still demand moderate-to-major human effort, implying diminishing marginal returns without parallel investment in specs/practices.
Cost–benefit and ROI drivers
- Upside: faster script authoring, higher throughput, and potential to reduce manual regression effort growth if scaled effectively.
- Costs/risks: reviewer time, refactoring/maintenance overhead from poorly structured AI outputs, investment in infrastructure (CI integration, traceability), and ongoing prompt/tuning costs.
- Realizable ROI requires spending on (a) improving specification clarity and standardization, (b) governance and review workflows, and (c) co-adaptation mechanisms (feedback loops, domain-aware evaluation agents).
Labor market & skills
- Demand shifts from routine script authoring to higher-value tasks: spec engineering, AI prompt engineering, semantic evaluation, and maintenance.
- Firms will need to retrain/test-engineering staff to manage and govern agentic tools, changing the mix of required skills.
Scaling and network effects
- Benefits grow with quality historical corpora: RAG improves as more high-quality spec→script pairs accumulate, creating positive returns to scale.
- However, there are negative externalities if low-quality AI outputs proliferate (increased maintenance costs, degraded trust), undercutting network benefits.
Policy & governance considerations
- Explicit human-in-the-loop constraints and artifact traceability reduce operational risk and liability—important for regulated industries.
- Standards for specification quality and AI evaluation will raise adoption efficiency and reduce hidden auditing costs.
Research & investment priorities for practitioners and economists
- Quantify total cost per automated test (generation + human review + maintenance) vs. manual automation baseline.
- Evaluate long-term maintenance burden induced by AI-generated code.
- Invest in tooling and organizational practices (spec templates, feedback pipelines) to increase the fraction of cases where AI yields net-positive automation.
- Study generalizability across domains and team sizes to better model firm-level adoption dynamics and labor impacts.

Overall, the study suggests agentic AI can increase productivity in test automation but will not, by itself, eliminate human labor or governance costs. Economic value depends strongly on complementary investments in specification quality, feedback loops, and review processes.

Assessment

Paper Typedescriptive Evidence Strengthlow — Single-company exploratory case study without a counterfactual or experimental design; measured improvements (throughput, code reuse) are plausible but may reflect selection, implementation, or measurement biases and limited external validation. Methods Rigormedium — Uses mixed methods (integration with CI logs, quantitative script metrics, and qualitative interviews/observations) and reports concrete operational metrics (e.g., 30–50% code reuse), but lacks randomized or quasi-experimental controls, pre-registration, or extensive robustness checks. SampleIndustrial deployment at Hacon (a Siemens company) across its Agile/Scrum engineering teams; evaluation draws on CI pipeline logs, generated candidate test scripts, code-reuse metrics, and qualitative data from developer reviews and interviews—single-organization sample with no external replication reported. Themesproductivity human_ai_collab GeneralizabilitySingle-company, domain-specific case (Hacon/Siemens) limits transferability to other firms or industries, Effectiveness depends on pre-existing quality of validated specifications — organizations without clear specs may see smaller gains, Integration and impact likely vary with CI tooling, tech stack, and team maturity in Agile practices, Results reflect an exploratory deployment and may not scale identically across larger or more heterogeneous teams, Findings may not generalize to other kinds of software engineering tasks beyond system-level regression test scripting

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
Automated regression testing is essential for maintaining rapid, high-quality delivery in Agile and Scrum organizations. Organizational Efficiency	positive	ability to maintain rapid, high-quality delivery	Reading fidelity high Study strength speculative	not reported 0.03
Validated test specifications accumulate faster than they are automated in many teams, limiting regression coverage and increasing manual work. Task Allocation	negative	regression coverage and manual testing workload	Reading fidelity high Study strength medium	not reported 0.18
We conducted an exploratory industrial case study of the Hacon Test Automation Copilot, an agentic AI system that generates system-level regression test scripts from validated specifications using retrieval-augmented generation and a multi-agent workflow. Developer Productivity	neutral	capability to generate system-level regression test scripts	Reading fidelity high Study strength medium	n=1 0.18
The Copilot was integrated with Hacon's CI pipelines and operates asynchronously as a 'silent AI teammate', producing candidate scripts for human review. Task Allocation	neutral	operational mode and integration with CI (asynchronous candidate generation for human review)	Reading fidelity high Study strength medium	n=1 0.18
Mixed-method evaluation shows the AI accelerates script authoring and increases throughput. Task Completion Time	positive	script authoring speed and throughput	Reading fidelity high Study strength medium	not reported 0.18
The Copilot achieves 30-50% code reuse when generating candidate test scripts. Developer Productivity	positive	code reuse in generated test scripts	Reading fidelity high Study strength medium	30-50% code reuse 0.18
Human review remains necessary for maintainability and correct domain interpretation of generated scripts. Output Quality	negative	maintainability and domain-correctness of test scripts	Reading fidelity high Study strength medium	not reported 0.18
Clear specifications, explicit governance, and ongoing human-AI collaboration are critical for successful scaling of regression automation. Governance And Regulation	positive	success of scaling regression automation / effectiveness of human-AI teaming	Reading fidelity high Study strength speculative	not reported 0.03
The paper provides lessons for scaling regression automation and enabling effective human-AI teaming in Agile settings. Training Effectiveness	neutral	availability of lessons and guidance	Reading fidelity high Study strength speculative	not reported 0.03