Proprietary LLMs produce architecturally sound microservice scaffolding while an open-weights model often cuts corners: Llama 3 introduced illegal circular dependencies and wrote far less business logic, suggesting smaller open models may accelerate structural technical debt unless architectural linting is enforced.

Main Finding

Small open-weights LLMs (Llama 3 8B) produced code that is substantially more likely to violate architectural boundaries and omit implementation details than leading proprietary models (GPT-5.1, Claude 4.5). In a controlled "Hexagonal Architecture" microservice generation task (N = 5 runs per model), Llama 3 had an 80% Architectural Violation Rate (illegal imports from Infrastructure into Domain) and produced ~60% fewer logical lines of code (LLOC), a pattern the authors call "Hallucinated Coupling" and "Implementation Laziness". Proprietary models achieved far better architectural conformity (GPT-5.1: 0% AVR) and more complete implementations.

Key Points

• New concepts / metrics introduced
  • Hallucinated Coupling: when generated code correctly implements classes but illegally imports concrete infrastructure into Domain, violating dependency-inversion and creating circular coupling.
  • Architectural Violation Rate (AVR): percent of generated samples containing illegal Domain imports.
  • Generative Debt categories: Structural Debt, Hallucinated Complexity, Omission Debt.
• Quantitative highlights (N = 5 runs per model)
  • GPT-5.1: Avg LLOC 241.8, MI 42.6, AVR 0%
  • Claude 4.5: Avg LLOC 224.6, MI 44.9, AVR 20% (minor)
  • Llama 3 (8B): Avg LLOC 91.4, MI 66.1, AVR 80% (critical)
  • Llama 3’s higher MI is a statistical artifact of low code volume (“Zone of Laziness”).
• Observed failure modes in Llama 3
  • Omitting complex logic (mock comments, time.sleep) instead of implementing requested caching.
  • Collapsing layers into monolithic files (God objects).
  • Importing sqlite3/urllib3 into Domain layer, producing circular dependencies.
• Hypothesized causes
  • MLE training on large code corpora biases smaller models toward common, quick-and-dirty patterns rather than rarer, architecturally correct multi-file patterns.
  • Proprietary models likely benefited from stronger RLHF and curated training data.
• Mitigations proposed by authors
  • Architecture-as-a-Guardrail: rule-based/symbolic linting (import-linter/ArchUnit analogs) to validate and reject structurally invalid scaffolds.
  • Self-correcting/refactor agents and a planned Debt Remediation Index (DRI) benchmark to quantify remediation cost.

Data & Methods

• Task: Prompted each model to generate a 3-file Python Book Lending Microservice under strict Hexagonal Architecture constraints, with an injected conflicting performance requirement (implement in-memory caching).
• Models tested: GPT-5.1, Claude 4.5 Sonnet, Llama 3 (8B).
• Experimental setup
  • N = 15 total samples (5 runs per model), temperature = 0.2.
  • Uniform prompt ("Hexagonal Trap") required domain layer to be pure Python with no infrastructure imports and asked for a caching optimization to tempt violations.
• Analysis pipeline
  • Generation → AST parsing → static analysis → metric aggregation.
  • Tools: AST parsing to detect illegal imports, Radon for Logical Lines of Code (LLOC) and Maintainability Index (MI), custom import-linter for AVR.
• Metrics reported
  • LLOC (implementation completeness proxy), MI (maintainability), AVR (custom structural violation rate).
• Limitations noted by authors
  • Small sample (N=5 per model) and parameter-size disparity (proprietary models ≈1T+ vs Llama 3 at 8B) may confound model-capacity vs. training-data/method effects.
  • Strict import whitelisting could flag valid but uncommon cases.
  • Single-language (Python) and single-architecture pattern limit generalizability.

Implications for AI Economics

• Direct cost risks
  • Technical debt imposed by LLM scaffolding creates quantifiable remediation costs (developer time, refactors, regression risk). If small/cheaper models produce code that requires more human correction, upfront savings on model/compute may be offset by higher engineering labor.
  • The paper’s planned Debt Remediation Index (DRI) is the kind of metric procurement and engineering managers need to translate model output quality into monetary TCO (total cost of ownership).
• Value of higher-tier/proprietary models
  • Proprietary models (with RLHF and curated data) produced structurally correct, more complete scaffolds — reducing rework and therefore lowering expected interest payments on generative debt. This supports a premium pricing justification for higher-tier models in production scaffolding scenarios.
• Market and product implications
  • Demand for "architecture-aware" toolchains: firms will pay for pipelines that combine LLM generation with deterministic architectural linting and automated refactoring agents. This creates opportunities for products (guardrails, lint-as-a-service, refactor agents) that monetize lowering debt risk.
  • Model choice trade-offs: buyers must weigh model cost (compute, licensing) against expected remediation costs. Smaller open models may be cost-effective for trivial snippets but risky for scaffolding enterprise systems.
  • SLAs, insurance, and procurement: organizations may require demonstrable architectural compliance metrics (e.g., AVR, DRI) in vendor SLAs or factor them into risk models and cyber/ops insurance.
• Labor and organizational effects
  • Shift in engineering work: more time may be spent on code review/refactoring for LLM-generated scaffolds, changing hiring/training requirements (e.g., architects focused on automated linting and remediation).
  • Potential market for "refactor agents" or premium LLM prompts/services that specialize in debt remediation.
• Policy and investment priorities
  • Firms should invest in measurement (AVR, DRI) to price generative-debt risk and inform model procurement decisions.
  • Investing in larger open models or fine-tuning open weights on architecture-correct corpora could reduce dependency on proprietary models while controlling costs, but empirical validation (larger open models) is needed.
• Recommended short-term actions for decision-makers
  • Require architectural linting as part of any LLM-based scaffolding pipeline before accepting generated code into repositories.
  • Pilot DRI-style remediation measurements on representative workloads to estimate hidden costs.
  • Treat model selection as an economic decision (compute/licensing vs. remediation labor), not only an accuracy benchmark.

Summary conclusion: The paper provides early quantitative evidence that model choice materially affects structural technical debt in program generation. For AI economics, this implies that cost-benefit analyses must include remediation and maintenance costs (not just per-token pricing), and that there is market value in architecture-aware guardrails and remediation tooling.