AI coding assistants do not check provenance before installing research software: in a pre-registered audit of 1,920 trials across six projects, assistants opened provenance signals in only 9 runs (0.5%) and none executed any verification, implying verification must be implemented in the runner rather than relied on model behavior.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
AI coding assistants now select, install, and configure software, and attackers have exploited that position through invented package names, compromised maintainer accounts, and manipulated repository text. In response, the supply-chain community publishes machine-checkable trust signals: software bills of materials, signed releases, build provenance attestations, and declared official channels. Whether coding assistants read or act on those signals has not been measured for any of these classes on research software. We pre-registered and ran a controlled study on six open-source research software projects (three HPC, three quantum computing) drawn from an 87-project corpus, with protocol, seed, panel, and analysis plan deposited with a DOI before any trial. W created nine modified copies for each project: no signal, one per signal class, two with a signature or attestation from the wrong issuer, one with all four signals, and one reproducing documented conflicts in the project's own metadata. Three models under two ways of operating an assistant, with and without an approval step, gave 1,920 registered trials, plus a supplement on three frontier models. We scored behavior from container logs rather than from what the assistant said, and recorded the cost of every trial. Verification was rare under every condition: in 9 of 1,920 registered trials (0.5%), the assistant opened any provenance signal before installing in 0 of 384 control trials, and no trial ran a verification command, so signal presence had no measurable effect. We drew three conclusions: publishing signals is necessary but not sufficient; price did not buy verification (the model that verified most often costs $0.10 per trial; the most capable, at $1.00, verified nothing); verification must be built into the program that runs the assistant. We release the per-trial cost ledger, the protocol, and every log.
Summary
Main Finding
AI coding assistants almost never read or verify published software-provenance signals before installing research software. In a pre-registered, controlled demand-side audit of 1,920 trials, assistants opened any provenance signal before installing in 9 trials (0.5%), and in none of the trials did an assistant run an actual verification command. Presence, absence, forgery, or contradiction of signals had no measurable effect on installation behavior.
Key Points
- Signal classes tested: SBOM, signed release, build-provenance attestation, and declared official distribution channels (SECURITY.md).
- Experimental scale: 1,920 pre-registered trials (6 real research projects × 9 conditions × 2 harnesses × 3 models with pre-specified trial counts), plus a 216-trial frontier-model supplement.
- Verification-positive trials: 9/1,920 (0.47%). No trial executed a verification command; none of the 384 control trials opened provenance before installing.
- Statistical result: signal presence had no measurable effect (registered fallback test p = 0.50).
- Cost accounting: every trial’s token use, USD cost (frozen rate card), and wall time were recorded. The model that checked most often cost $0.10 per trial; the most capable, $1.00-per-trial model, did no verification.
- Two harness modes: gated (requires human approval before each command) and autonomous (runs commands without approval). Neither harness reliably produced verification.
- Pre-registration and reproducibility: full protocol, seeded draw, tooling, per-trial logs, and cost ledger were deposited and released; one registered local model was replaced after diagnostics and that deviation was logged.
- Outcome measurement relied on container file-access and command logs, not assistant self-reports.
Data & Methods
- Panel selection: 6 research-software projects (3 high-performance computing, 3 quantum) drawn by pre-seeded, screened procedure from an 87-project corpus; screening rules S1–S5 ensured reachable, installable-from-source projects and diversity.
- Conditions (9 per project): control (unmodified), sbom_present, signed_release_present, signed_release_issuer_mismatch, attestation_present, attestation_issuer_mismatch, channel_declaration_present, all_signals_present, inconsistent_surface (reproduced real metadata conflicts).
- Signal injection: created signed releases (OpenPGP), in-toto/SLSA-format attestations (DSSE envelope), CycloneDX SBOMs, and SECURITY.md channel declarations; wrong-issuer cases used signatures/attestations issued under researcher-controlled identities to emulate forgeries/hijacks.
- Harnesses and models:
- Harnesses: gated (approval step before commands) and autonomous.
- Main-study models: qwen2.5-coder:14b (local reproducible), claude-sonnet-5 (production-realistic), gemini-3.5-flash-lite (cost-accessible).
- Frontier supplement: claude-fable-5, gpt-5.6, Kimi-K2.6 (smaller descriptive supplement on two extreme conditions).
- Execution environment: each trial ran in an isolated Docker container with networking disabled; assistants installed from the local modified repository copy (no registry/network access).
- Trials and coding: 3 trials per cell on hosted models, 10 per cell on local model (doubled for shared control) for a total of 1,920 registered trials. Event coders labeled actions from container logs (file reads, commands run, exit statuses). Outcomes recorded as proceeded / declined / verified-then-proceeded / incomplete.
- Transparency: protocol, tooling, modified repository generators, raw transcripts, uncoded coding sheet, deviations log, and full per-trial cost ledger released.
Implications for AI Economics
- Supply-side signals alone do not change demand-side behavior. Investments by maintainers to publish SBOMs, attestations, and signed releases create social value only if consuming tools (here, AI assistants/harnesses) are designed to read and act on them. This is a coordination failure: private incentives to publish may not translate into reduced risk unless platform and tool vendors internalize verification.
- Regulatory limits: mandates (for example, SBOM requirements) that focus on supply-side disclosure may have limited effect on actual security outcomes unless compliance is paired with requirements or incentives for verifier adoption at the tool/harness/platform level.
- Market failure and externalities: downstream consumers (researchers, organizations) expect assistants to check provenance but currently do not get that service. This generates negative externalities (vulnerable installations, supply-chain exploitation) that neither maintainers nor model providers alone internalize.
- Price/performance mismatch: a low-cost model sometimes checked slightly more often, and the most expensive/capable model did no verification — meaning market competition on model capability or price does not reliably produce verification behavior. Buyers cannot simply pay more for an assistant and expect provenance checks.
- Productization and business opportunities: there is demand for verification-as-a-service, verified-harness products, or certified-agent platforms that integrate provenance checking into the assistant runtime. Firms could monetize reliable verification toolchains integrated with package managers and agent harnesses.
- Procurement and liability: institutions should specify provenance-verification requirements in procurement of assistant services and consider liability frameworks that push platform vendors to adopt verification-by-default. Certification programs or standard APIs for agent verification could align incentives.
- Policy recommendations for effectiveness:
- Require or incentivize verifier integration in agent runtimes and package managers (not only supply-side disclosures).
- Standardize verification UX in harnesses (approval workflows should surface provenance checks, not only be a gate for commands).
- Fund/mandate open verification tooling and audits, and require per-install verification logging for accountability.
- Research-software specifics: because the study focused on research software installed from source (not registry-based installs), results highlight risks in academic and specialist ecosystems where agents still manage installs; transferability to mainstream registry installs is plausible but needs direct measurement.
Limitations (brief) - Small panel (6 projects) drawn from HPC/quantum research software; behavior on mainstream package ecosystems (npm/pypi) may differ. - Containers were network-disabled and installations were from local checkouts; some remote checks (e.g., checking a remote transparency log) were outside the experimental setup by design, though local verification of provided artifacts was possible. - The audit measured current assistant behavior under specific harnesses and prompts; different prompt engineering, harness designs, or model updates could change outcomes.
Bottom line: publishing SBOMs, signatures, and attestations is necessary but not sufficient. To get provenance checks in practice, verification must be integrated into the software that runs assistants and into procurement/regulatory incentives — otherwise the market won’t automatically make installations safer.
Assessment
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| AI coding assistants opened a provenance signal before installing software in only 9 of 1,920 registered trials (0.5%), and they ran no verification command in any trial. Regulatory Compliance | negative | Opening and cryptographically or procedurally verifying SBOMs, signed releases, attestations, or channel declarations before installation |
Reading fidelity
high
Study strength
medium
|
n=1920
0.5% of trials opened any provenance signal; 0 trials ran a verification command
|
| The presence of provenance signals had no measurable effect on assistant verification behavior. Regulatory Compliance | null_result | Assistant verification behavior in response to the presence of provenance signals |
Reading fidelity
high
Study strength
medium
|
n=1920
p = 0.50
|
| Assistants did not read provenance signals regardless of whether the signals were valid, forged, combined, or contradicted by the project’s own metadata. Regulatory Compliance | negative | Assistant reading or acting on provenance information across valid, invalid, composite, and contradictory conditions |
Reading fidelity
high
Study strength
medium
|
n=1920
|
| Adding an approval step to the assistant harness did not supply provenance verification. Regulatory Compliance | negative | Whether an approval-gated assistant verified provenance before installation |
Reading fidelity
high
Study strength
medium
|
n=1920
0 verification commands in any trial
|
| The model that verified most often cost US$0.10 per trial, while the most capable model cost US$1.00 per trial and verified nothing. Regulatory Compliance | null_result | Verification frequency as a function of per-trial model cost |
Reading fidelity
high
Study strength
low
|
n=1920
US$0.10 per trial versus US$1.00 per trial
|
| The study used six research-software projects selected from an 87-project corpus: three high-performance-computing projects and three quantum-computing projects. Other | other | Scope and composition of the evaluated research-software panel |
Reading fidelity
high
Study strength
medium
|
n=87
|
| The experiment was pre-registered before data collection, including the hypothesis, project-screening rules, condition matrix, exclusion and stopping rules, and cost-accounting scheme. Governance And Regulation | other | Pre-registration and prospective specification of the experimental design |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Assistants referenced an official-channel statement in none of 60 observational pilot decision traces. Regulatory Compliance | negative | References to official-channel provenance statements during assistant decision-making |
Reading fidelity
high
Study strength
low
|
n=60
0 of 60 decision traces
|