Enterprises that default to the largest LLM risk unnecessary emissions without quality gains: a 520-task supply-chain benchmark finds smaller or differently architected models can match accuracy while producing orders-of-magnitude less per-task carbon, and a carbon-aware procurement framework plus calibrated routing cuts estimated inference carbon by roughly 98% while preserving or improving decision quality.

Main Finding

Benchmarking six LLMs on 520 supply-chain tasks shows that larger nominal model size is an unreliable proxy for decision quality, and that substantial variation in operational (inference) carbon exists across models. A TOE-grounded Carbon-Aware AI Procurement Framework (CAAPF) — operationalized as a category×tier calibrated routing policy (GreenRoute) — can deliver materially higher quality per unit carbon than naïve “choose the biggest model” procurement. In this sample, the Pareto frontier is formed by Claude Haiku (low carbon, near-top quality) and Claude Sonnet (highest quality, moderate carbon); a simple low-carbon static choice or a calibrated routing policy is often preferable depending on an organization’s quality threshold.

Key Points

• Models evaluated (aggregate score on 520 tasks):
  • Claude Sonnet 4.6: score 0.723; Tier‑1 GT 78.6%
  • Claude Haiku 4.5: score 0.699; Tier‑1 GT 75.3%
  • Mistral Large 3: 0.613
  • Llama 3.3 70B: 0.528
  • Llama 4 Scout: 0.521
  • DeepSeek R1: 0.497
• Estimated generation-related per-task carbon (study assumptions; ±50% uncertainty):
  • Haiku: 0.022 gCO2/task (312 tokens/task)
  • Scout: 0.023 gCO2/task (387 tokens/task)
  • Sonnet: 0.401 gCO2/task (445 tokens/task)
  • Llama 3.3 70B: 0.468 gCO2/task (478 tokens/task)
  • Mistral Large 3: 3.427 gCO2/task (623 tokens/task)
  • DeepSeek R1: 17.082 gCO2/task (2,847 tokens/task)
• Verbosity amplification effect: per-token carbon rates understate workload-level emissions when models generate vastly different output lengths (DeepSeek generates ~9× more tokens than Haiku and produces far higher per-task emissions despite lower accuracy).
• SC-domain knowledge (30 “trap” tasks): Claude models substantially outperform others (Sonnet 0.893, Haiku 0.861), indicating domain-relevant differences not captured by parameter counts.
• CAAPF principle: “benchmark first, select green.” Steps: define task portfolio & thresholds (τ), benchmark candidates, measure quality/carbon/cost/latency, filter by admissibility, select lowest-carbon admissible model, monitor & re-evaluate.
• GreenRoute (proof of concept routing calibrated per category×tier with τ=0.65):
  • Out-of-sample mean quality 0.733 (SD 0.004)
  • Mean estimated carbon 0.402 gCO2/task (SD 0.124)
  • Estimated savings vs. always-DeepSeek: 97.6%
  • Choice logic: route only when task‑level heterogeneity improves the feasible frontier; otherwise prefer static lowest-carbon passing model.
• Propositions:
  • P1: Model size is not a reliable procurement proxy when jointly considering quality and operational carbon.
  • P2: Preferred selection strategy depends on organization-specific quality/risk thresholds; routing is not always beneficial.
  • P3: A TOE-based (Technology‑Organization‑Environment) process creates an auditable procurement record incorporating environmental pressures without making carbon the sole criterion.

Data & Methods

• Benchmark:
  • 520 supply-chain tasks across 6 categories: Demand Forecasting (92, incl. 33 M5 tasks), Vehicle Routing (84), Inventory Optimization (96), Supplier Risk Assessment (88), Order Fulfillment (80), Demand Classification (80).
  • Three difficulty tiers: Tier 1 (formula/deterministic ~35%), Tier 2 (multi-factor ~40%), Tier 3 (strategic judgment ~25%).
  • 30 SC‑trap tasks specifically probing supply-chain domain knowledge.
  • Tasks authored and validated internally (three-stage validation), but not validated by external practitioners.
• Models: six availability‑based LLMs from four providers (sample concentrated among a few vendors; Anthropic models lacked disclosed parameter counts).
• Prompting & evaluation:
  • Single prompt protocol, model temperature = 0, output caps 1,024–2,048 tokens.
  • Dual LLM-as-judge evaluators: Claude Opus 4 and Qwen3-235B; scored numeric accuracy, reasoning quality, and domain knowledge (1–10 scales), averaged and normalized.
  • Tier-1 tasks validated against deterministic ground truth (exact match within 5%) to control for judge bias.
• Carbon estimation:
  • Model-specific gCO2/MTok coefficients computed from serving hardware, GPU power, throughput, PUE (1.1–1.2), and a US‑East grid intensity assumption (0.38 kgCO2/kWh); estimates carry ±50% uncertainty.
  • Per-task carbon = gCO2/MTok × mean output tokens / 1e6. Input/prefill tokens not included (so values reflect output-generation carbon only).
• GreenRoute:
  • Calibrated per category×tier (18 cells); cell-level τ = 0.65 used to select lowest-carbon model that meets threshold (else highest-quality).
  • Evaluation: 20 repeated stratified 50/50 split-half trials (seed 42); reported out‑of‑sample metrics.

Limitations noted by authors: - Sample not representative of global LLM market (vendor concentration and disclosure gaps). - Carbon estimates are engineering assumptions, approximate and output-only (±50%). - Single prompting protocol; did not explore demonstrations, prompt engineering, or temperature effects. - Judges are LLMs (possible evaluator biases) and provider defaults differ slightly. - Confounding across provider, architecture, training data; causal claims about why differences arise are limited. - GreenRoute proof-of-concept does not evaluate production metadata classification costs, routing overhead, or real-world governance integration.

Implications for AI Economics

• Procurement heuristics: The common “choose the largest model” rule can be economically and environmentally suboptimal. Organizations should benchmark domain-specific task quality and account for operational carbon when procuring models.
• Internalizing energy externalities: Including estimated inference carbon in procurement decisions (and reporting) creates incentives for providers to optimize inference efficiency and for buyers to choose lower-carbon models when acceptable quality is achieved.
• Cost-quality-carbon tradeoffs: Organizations with moderate quality requirements can often pick a low-carbon static model (e.g., Haiku) and obtain large emissions reductions for little quality loss. High-quality requirements may justify higher-carbon choices (e.g., Sonnet) or calibrated routing that mixes models across task classes.
• Routing economics: Task-level routing can raise system complexity and overhead; it is only justified if it improves the feasible quality–carbon frontier given organization-specific thresholds. The paper’s CAAPF operationalizes the decision rule (benchmark first; route only if needed).
• Market signals & provider incentives: Empirical procurement that rewards low-carbon, high-quality inference will pressure providers to disclose per-inference energy metrics, invest in more efficient architectures, or offer differentiated pricing for efficient endpoints.
• Policy & governance: TOE-informed, auditable procurement records that include carbon measurements align with SDG 12/13 and enable compliance reporting, internal carbon accounting, and regulated procurement standards.
• Research & measurement needs: Better standardized per-request energy/carbon reporting from providers, end-to-end serving measurements (including input/prefill tokens and infrastructure variability), and evaluations across more diverse and representative models and prompting protocols are required for robust economic policy design.

Bottom line: For enterprise supply-chain AI, benchmarking domain-specific decision quality together with operational carbon yields better procurement choices than naive size-based rules. CAAPF offers a practical, auditable decision sequence — “benchmark first, select green” — that firms can adopt to align AI deployments with economic and environmental objectives.