Most AI-generated test files do little verification — 80% lack strong assertions — but when tests contain explicit oracle signals PRs are noticeably more likely to be accepted, increasing merge odds by about 28% after controls.

Main Finding

Agent-authored test files largely lack meaningful test oracles: across 86,156 cumulative test-file patches from 33,596 agent-authored PRs, 80.2% contain weak or no explicit oracle signals (i.e., no or only execution-level checks). Although raw merge rates look lower for PRs with the strongest oracle signals, a multivariate regression that controls for agent, PR size, repo popularity, task type, and language finds that PRs with multi-strong oracles (S3) are more likely to be merged (OR = 1.28, p < 0.001). In short, agents reliably generate test structure but frequently fail to generate substantive verification logic, and oracle strength is an independent positive signal for merge likelihood once confounders are controlled.

Key Points

• Scope and scale
  • 86,156 cumulative test-file patches from 33,596 agent-authored pull requests across 2,807 GitHub repositories (AIDev-pop subset).
  • Five agents analyzed: OpenAI Codex (dominant, ~65% of PRs), GitHub Copilot, Devin, Cursor, Claude Code.
• Oracle taxonomy (syntactic, regex-based)
  • Eight oracle-signal categories: Weak (W1–W5) to Strong (S1–S3).
    • W1: no assertion; W2: existence/non-null checks; W3: boolean asserts only; W4: mock/call verification; W5: snapshot match only.
    • S1: value equality/comparison; S2: error/containment/type checks; S3: two or more distinct strong types.
• Prevalence and agent differences
  • 80.2% of test-file patches are weak/no-oracle (W1–W5).
  • Strong-oracle presence overall: S1 = 11.3%, S3 = 5.7%.
  • Newly created test files show higher strong-oracle rates; across agents the new-file strong rate ranges ≈18% to 67% (agent-dependent).
  • Agents differ significantly in oracle quality (χ2 test reported).
• Merge outcomes and review effort
  • Raw rates: S3 PRs merged at 59.7% vs 72.6% for weak-oracle PRs, but S3 PRs are much larger, more reviewed, and in more popular repos.
  • After controlling for confounders (agent, PR size, repo stars, task type, language), S3 oracles significantly increase merge likelihood (OR = 1.28, p < 0.001).
  • S3 PRs attract 2.4× more review effort and ~4.2× more code additions on average.
• Validation and limitations
  • Human labeling: stratified sample of 384 patches, Cohen’s κ = 0.77; automated classifier matches human labels in 86.7% of sampled patches.
  • Limitations include regex/syntactic approach (may miss implicit or pre-existing oracles), dataset filtered to repos with ≥100 stars, timeframe and agent mix may limit generalizability.

Data & Methods

• Data source: AIDev-pop subset (AIDev dataset).
• Test-file identification: path-based test patterns + 30 source-code extensions; non-executable artifacts excluded.
• Cumulative patch construction: concatenated diffs per (PR, filename) in commit order to capture the full oracle signal introduced by the PR (86,156 cumulative patches from 103,976 initial test patches).
• Oracle signal taxonomy developed via iterative open coding; applied regex-based detectors for common assertion patterns across major testing frameworks (pytest, JUnit, Jest, Go testing, RSpec).
• Manual validation: 384 stratified random patches labeled by two authors; disagreements resolved and inter-rater reliability computed.
• PR-level aggregation: each PR assigned its strongest oracle-signal class across its test-file patches; grouped into three PR quality groups (all weak, has strong, has multi-strong).
• Outcomes measured: merge outcome, review effort (formal reviews + inline + discussion comments), PR size (files changed, lines added), repo stars, time-to-close.
• Analyses: descriptive distributions, controlled comparisons within agents/task types/PR size buckets, and multivariate logistic regression controlling for agent, log(PR size), log(repo stars), task type, and primary language.

Implications for AI Economics

• Measurement and productivity metrics
  • Current productivity or quality metrics that count presence of tests (or coverage) will overestimate verification value when agent-authored tests are included. Firms relying on naïve KPIs (test-file counts, coverage badges) risk overstating agent productivity and software quality.
  • Economic evaluations of agent impact (labor substitution estimates, ROI models) should discount the verification value of agent-generated tests unless oracle quality is measured.
• Incentives and tool procurement
  • Buyers and teams should prefer agents or configurations that are explicitly optimized for oracle generation (agents differ markedly on oracle rates). Procurement decisions and pricing models could reflect differences in verification capability, not only code-completion throughput.
  • Vendors could monetize higher-quality oracle generation (or offer it as a premium feature) because strong oracles correlate with higher merge likelihood after adjustment.
• Cost structure and reviewer labor
  • Agent-generated PRs with stronger oracles are associated with larger, more-reviewed PRs; agent adoption may shift reviewer workloads rather than simply reduce them. Economic models of labor substitution must account for increased review and integration effort.
  • Organizations may incur additional costs to implement oracle-aware CI checks, manual audits, or curated prompting strategies that improve oracle quality.
• Risk, externalities, and downstream costs
  • Weak or absent oracles create “test theater” that masks correctness issues; undiscovered faults can impose downstream debugging, incident, or reputation costs. Economic assessments of agent deployment should include probabilistic estimates of such downstream failure costs.
• Policy and governance
  • Firms and regulators should consider standards for verification signals in automated contributions (e.g., require assertion patterns for newly added test files, or flag test files lacking oracles for manual review).
  • Procurement and approval workflows (e.g., for third-party code or agent-generated PRs) can include oracle-aware gates to reduce hidden risk and better align incentives.
• Recommendations for economic actors
  • Update KPIs to include oracle-aware measures (e.g., fraction of test files with S1–S3 syntactic signals; follow-up validation or mutation testing success rates).
  • Differentiate agents on verification capability in vendor evaluations and cost-benefit analysis.
  • Invest in complementary labor (test-focused reviewers, mutation-guided validation) and in training/prompt engineering that explicitly requests behavioral assertions.
  • Track downstream incidents and remediation costs attributable to weak oracles to refine ROI and deployment strategies.

Concluding note: the study shows a measurable verification gap in agent-authored tests that alters the economic calculus of adopting coding agents — agents produce test structure cheaply but often not substantive verification, so organizations should recalibrate productivity, procurement, and governance decisions to reflect oracle quality rather than test-file counts alone.