Pervasive AI let a six-student team deliver a 21k-line RAG-based onboarding tool with just 35 human hours and US$69 of tooling, which the authors estimate would have taken ~329 professional hours — a corrected ~9.9× cost ratio that underlines how easy it is to mismeasure AI development costs.

Main Finding

A six-person student team built a RAG-based conversational onboarding assistant in one academic term with pervasive AI help. Using a three-layer cost model (actual AI billing, logged human hours, and a retrospective human counterfactual), the authors estimate the project cost with AI at US$304 versus a counterfactual of US$3,005 — a corrected cost ratio of ≈9.9× (counterfactual / actual). The primary contribution is methodological: two easy-to-make, often-invisible measurement errors (assuming per-token billing when the tool was on a flat subscription; and using inappropriate regional wage rates) initially inflated the ratio to 19.4×. Correcting these demonstrates how fragile cost claims can be and argues for more rigorous costing practices in AI-assisted development.

Key Points

• Artifact and scope: ~21k production LOC, ~25 features, 201 automated tests, multi-language RAG pipeline, frontend (React/TS) + backend (Python/FastAPI), vector store (ChromaDB), configurable LLM providers.
• Team/process: six students, four phases (problem framing, solution design, build-and-test, measurement), two AI tools (frontier LLM for early design, a mid-tier coding agent for implementation).
• Three-layer cost model:
• Real AI spend from billing (flat-rate subscription + metered repo-analysis + embeddings).
• Self-reported human effort (35.2 hours logged).
• Human counterfactual (retrospective estimate of hours a professional would need; 329 hours priced at regional hourly rates).
• Core numbers (corrected):
  • Tooling (billing): US$69 total.
  • Logged human effort valued at mid-level rate (US$6.67/h): US$235.
  • Total actual project cost: US$304.
  • Counterfactual cost: US$3,005 → ratio ≈9.9×.
• Initial errors that produced 19.4×:
• Inferring per-token cost from token counts for an agent that ran on a flat monthly subscription (token volume did not affect spend).
• Applying national/metropolitan wage rates rather than regional rates (inflated counterfactual by ~85%).
• How AI was used (self-assessed % assistance):
  • Code writing & test generation ≈95% AI-assisted.
  • Debugging ≈80%.
  • Documentation ≈85%, requirements ≈70%.
  • Prompt design ≈40%, architectural decisions ≈20%.
• Practical lessons: verify provider pricing before optimization, manage long-agent session context explicitly, prefer specification-first workflows, treat prompts as versioned artifacts, and enforce upfront security rules.

Data & Methods

• Case study design: an instrumented student project with time logs, billing receipts, prompts, and a development log. Replication package includes redacted billing, prompts, logs.
• Three-layer cost model applied per project phase:
  • Tools: taken from provider billing records (Copilot Business subscription apportioned; metered repo analysis; embeddings).
  • Human: hours self-reported by team members and valued at mid-level regional hourly rate (US$6.67/h).
  • Counterfactual: bottom-up retrospective estimate of professional hours needed per task without AI, priced using regional hourly wages (junior/mid/senior US$3.86/6.67/10.88 from local gross salaries).
• Sensitivity and correction process: initial token-count pricing replaced with actual billing dashboard figures; counterfactual re-priced using local rather than national salary data. Authors emphasize remaining uncertainty in the counterfactual (retrospective, not experimentally measured).
• Limitations: single-case, student team, self-reported effort and counterfactual hours, not a controlled productivity experiment — the ratio is an order-of-magnitude signal, not a general benchmark.

Implications for AI Economics

• Measurement rigor is critical: common, subtle errors (misreading pricing models; misapplying wage data) can substantially bias estimates of AI’s cost-saving or productivity effects. Researchers and practitioners must verify billing and local labor assumptions before reporting dollar benefits.
• Recommend standardized reporting for AI-assisted development studies:
  • Provide billing records or clear provider-pricing reconciliation.
  • Distinguish flat subscriptions vs metered pricing and apportion subscriptions explicitly.
  • Report logged effort separately from counterfactual estimates and disclose how counterfactual rates were chosen (region, seniority).
  • Include sensitivity analyses showing how results change with wage and pricing assumptions.
• Economic interpretation: AI in this case reallocated tasks rather than replaced headcount — implementation tasks were highly delegable, but judgment and architectural roles remained human. Policy and firm-level analyses should therefore treat AI as shifting task composition (and required skills) more than simply reducing labor demand one-for-one.
• Implications for productivity measurement: headline multipliers (e.g., “10× faster/cheaper”) are fragile; economists should treat single-case multipliers as preliminary signals and prefer aggregated, audited studies across projects and contexts.
• Research agenda: replicate the three-layer costing approach across more projects (different sizes, industries, professional teams) to estimate heterogeneity in multipliers; develop standardized protocols for counterfactual elicitation or experimental designs (randomized trials, matched controls) to reduce retrospective bias; and study substitution vs complementarity at the task level to forecast labor market impacts more accurately.

Concise recommendations for researchers/practitioners: - Always check provider billing dashboards; do not infer costs from token counts when subscription plans may apply. - Record metered charges in real time and apportion subscriptions transparently. - Use region-appropriate wage rates and report sensitivity ranges. - Separate and publish logged effort and counterfactual assumptions; make replication artifacts available when possible.