LLMs produce IaC with substantially higher static-analysis warnings than humans: across models and vendors, generated templates show about 3.2–3.9× the vulnerability density of size-matched human templates, with the largest relative deficits on the simplest tasks; vendor 'extended-thinking' cuts flagged issues by ~12% but is only marginally engaged (under 1% of tokens), limiting practical impact.

Main Finding

When compared to a size-matched human baseline, LLMs authoring Infrastructure-as-Code (IaC) produce templates with substantially higher vulnerability density: across 12 model configurations every model lies between 3.21× and 3.87× the human vulnerability density (size-matched). The gap is largest on the simplest tasks (4.9× at single-resource files) and shrinks with artifact size (≈1.4× at ≥20 resources). Vendor “extended-thinking” APIs yield a modest but statistically significant reduction in vulnerability density (≈−12% vs prompted chain-of-thought, p = 0.0013), while prompt-engineered chain-of-thought alone provides no measurable security benefit versus standard generation.

Key Points

• Human baseline: 634 human-authored IaC templates scanned with identical toolchain; mean density 2.66 findings/resource (median 1.00).
• Benchmark: GenIaC-SecBench — 100 natural-language deployment scenarios (60 simple, 40 complex) → 1,196 generated artifacts scanned.
• Scanners: three independent policy engines used—Checkov, KICS, Trivy—total findings = 38,803.
• Size matters: vulnerability density strongly inversely correlates with declared-resource count (Spearman ρ = −0.55, p ≈ 1.7×10−78). Unmatched comparisons confound security with artifact size.
• Aggregate model gap: size-matched generated IaC is 3.21×–3.87× worse than humans (per-resource density).
• Complexity interaction: gap largest on single-resource/simple tasks (4.9×) and declines with more resources.
• Reasoning decomposition:
  • Three arms tested on same base model: standard generation, prompted chain-of-thought (CoT), vendor extended-thinking API.
  • Extended-thinking vs prompted CoT: ~−12% density (p = 0.0013).
  • Prompted CoT vs standard: no significant difference.
  • Reason: extended-thinking uses very few reasoning tokens on IaC (median 29 tokens simple, 151 complex), under 1% of typical completion length on complex tasks—so the mechanism is only lightly engaged.
• Methodological contributions:
  • Introduces size‑matched human security baseline for IaC model evaluation.
  • Uses Skillings–Mack statistic (generalized Friedman) because realistic multi-model benchmark designs produce incomplete blocks and Friedman is uncomputable.
  • Calibrates an LLM judge against human experts and releases full code/data.
• Negative/nuanced findings:
  • No evidence that more deployable models are systematically more vulnerable (r = 0.158, p = 0.625).
  • Some apparent low absolute finding counts (e.g., phi3) are survivorship bias: models that output few parseable resources look safer by count but are worse per declared resource.

Data & Methods

• Scenarios: 100 natural-language deployment tasks (60 simple single-service, 40 complex multi-component), covering AWS/Azure/GCP/provider-agnostic Kubernetes and four IaC formats (Terraform HCL, CloudFormation, ARM, Kubernetes manifests).
• Models/configurations: 12 configurations across four vendors (Anthropic, OpenAI, Google, open local models), including three reasoning-mode isolates (standard, prompted CoT, vendor extended-thinking).
• Generation protocol: single stateless API call per scenario; system prompt sets role (senior cloud infra engineer); no few-shot, no security instructions; temperature 0.2 for standard/CoT, vendor defaults for extended-thinking; very large token ceilings (truncated responses discarded).
• Validation/scanning: schema validation per format (terraform validate, cfn-lint, ARM parsing, kubeconform), then scanned by Checkov, KICS, and Trivy; complete coverage verified for all generated artifacts.
• Human corpus: 634 public, production-style IaC templates filtered and scanned with identical toolchain for a size-matched baseline.
• Primary metric: vulnerability density = (total findings across three engines) / (declared-resource count from AST).
• Statistics:
  • Use Skillings–Mack for omnibus tests due to incomplete-block design.
  • Post-hoc: Wilcoxon signed-rank with Holm correction; Mann–Whitney U for human–model comparisons.
  • Count modeling: negative-binomial GEE with log(resource count) exposure offset to produce incidence-rate ratios (handles overdispersion; variance-to-mean ≈ 130).
• Key quantitative results:
  • 1,196 generated artifacts scanned; 38,803 findings.
  • Size-matched model-to-human density ratio band: 3.21×–3.87×.
  • Single-resource ratio: 4.9×; ≥20-resources ratio: ~1.4× (not significant at p=0.058).
  • Extended-thinking effect: ~−12% density vs prompted CoT (p = 0.0013); vs standard −13.2% (p = 0.012).
  • Reasoning-token spend under extended-thinking: median 29 tokens (simple), 151 (complex) against completions of ~886 and ~18,533 tokens respectively (<1% on complex).

Implications for AI Economics

• True security externalities of LLM-assisted IaC are nontrivial: per-resource vulnerability density is several× higher for model-generated artifacts versus human-authored counterparts, implying increased expected costs from remediation, incident response, and potential outages. Economic evaluations of LLM deployment should account for these downstream security costs, not just productivity gains.
• Pricing vs benefit of "reasoning" features: vendor extended-thinking APIs produce measurable security improvements (~10–15%) but consume few reasoning tokens in practice on IaC tasks. The marginal security benefit per paid reasoning-token is small for this task class, so buying expensive reasoning capacity may yield limited ROI unless vendors change how tokens translate to computation/engagement.
• Procurement and policy: purchasers should demand size-matched, human-anchored security benchmarks when comparing assistant models, and should require rigorous, comparable scanning regimens (multiple engines) to avoid tool-specific illusions of safety. Contracting and SLAs should reflect per-resource security performance, not raw output counts.
• Model selection and market differentiation: across many frontier models and vendors, size-matched vulnerability densities fall in a surprisingly narrow band (3.21×–3.87× human). This suggests limited current differentiation on default-security posture for IaC; economic competition may instead focus on pricing, developer UX, or add-on safety tooling.
• Incentives for toolchains and vendors: since models tend to over-generate resources on simple tasks and default to insecure settings (networking/IAM), there is market demand for rule-aware generation, automated post-generation sanitizers, or security-aware prompting/filters. Investing in such layers could reduce downstream security costs and become a monetizable product.
• Methodology for economic analyses: researchers and practitioners should size-match comparisons (control for declared-resource counts), correct for survivorship bias (models that fail to emit parseable resources can distort counts), and adopt statistics suitable for incomplete-block designs (Skillings–Mack). These are necessary to produce valid cost-benefit estimates of LLM adoption in software infra contexts.
• Policy/regulatory angle: regulators evaluating safe deployment of LLMs in critical infrastructure should require human-anchored baselines and explicit measurement of per-resource risk increases, not only absolute vulnerability counts or anecdotal fixes.

All code, data, and regeneration scripts for the benchmark are publicly released by the authors, enabling replication and further economic modeling of the security externalities reported.