A new benchmark reveals frontier LLMs struggle to assemble auditable product carbon footprints: direct single-shot estimates fall within 2× of declared EPD totals 60–77% of the time, but compositional, step-by-step pipelines succeed on only 37–58% and often violate mass conservation, undermining transparency for product-level decarbonization.

Main Finding

PCFBENCH is a new, expert-annotated diagnostic benchmark that decomposes cradle-to-gate product carbon-footprint (PCF) estimation into six independently-evaluable tasks (decompose, triage, map, material-rate extraction, energy-rate extraction, aggregate/validate). Across 614 labeled items, eight frontier LLMs and one specialist baseline show much weaker performance when required to produce inspectable, compositional PCFs than when asked to output a single aggregate kgCO2e number. Direct single-call predictions reach 60–77% of products within 2× of declared EPD totals; the compositional, step-by-step pipeline achieves only 37–58% within 2×. Key failure modes are missing BOM components, incorrect ontology mappings, numerical/extraction errors, and violations of basic mass-conservation constraints — problems that aggregate scoring can hide via error cancellation.

Key Points

• Benchmark scope and size

  • PCFBENCH: 614 expert-labelled items across six tasks.
  • Task breakdown: Decompose (94 products), Triage (200 items), Map (109 materials), Material extraction (22 questions / 55 claims from 36 PDFs), Energy extraction (14 questions / 34 claims), Total kgCO2e validation (175 EPDs).
  • Dataset and code released: https://huggingface.co/datasets/Watershed-Climate/PCFBench and https://github.com/watershed-climate/pcfbench.

• Evaluation design

  • Two operational modes: Direct (single LLM call returns kgCO2e) and Compositional (chain Task 1–5 agent outputs, deterministic aggregation).
  • Task-specific metrics (e.g., F1 for decomposition/extraction, exact-match for mapping, claim-F1, and within-2× and median |RE| for total kgCO2e).
  • Ground truth: verified Environmental Product Declarations (EPDs) and curated evidence from technical PDFs.

• Baselines and headline results

  • Eight frontier LLMs from four providers plus Parakeet (mapping specialist) were benchmarked (examples: GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.6).
  • Direct total-kgCO2e: 60–77% of products within 2× of declared EPD totals.
  • Compositional (chained) total-kgCO2e: only 37–58% within 2×.
  • Decomposition: recall ≈ 0.64–0.75, precision ≈ 0.85–0.92 (i.e., models often omit 25–35% of expert-listed components).
  • Mapping: specialist Parakeet scores ~0.76 (mapping EM reported), best general LLMs can match or exceed it on some items (e.g., Gemini 3.1 Pro high mapping EM), but many domain-specific mapping errors persist.
  • Extraction and numerical reasoning are weak (claim-F1s low; extraction hallucinations occur).
  • Physical-consistency failures: 25–55% of compositional BOMs violate mass conservation; 47–86% include at least one “ghost” (zero-mass) component.

• Notable error modes

  • Under-specification and un-stated assumptions (LCA decisions depend on scope/method; LLMs often commit without documenting assumptions).
  • Ontology mismatches: lexical similarity ≠ domain-correct mapping (e.g., selecting gold rather than a plating chemical).
  • Error cancellation: correct aggregate kgCO2e can mask compensating upstream mistakes.

Data & Methods

• Tasks and schemas

  • Task 1: Decompose product description into bill-of-materials (BOM) entries.
  • Task 2: Triage — decide whether a BOM item maps directly to a background database activity or must be decomposed.
  • Task 3: Map — select the correct ecoinvent (v3.11 subset) activity for each mappable input; set includes 2,574 market activities as options.
  • Task 4: Material-rate extraction — read PDFs (EPDs, reports, papers) and extract kg per declared unit with evidence.
  • Task 5: Energy-rate extraction — extract electricity/heat/fuel per declared unit and judge scope overlaps with background processes.
  • Task 6: Deterministic aggregation (multiply rates × EFs, sum) — not LLM-evaluated.
  • Task 7: Validate total kgCO2e against EPD-declared totals (175 items; progressive disclosure settings: name, +description, +composition, +region).

• Annotation and provenance

  • Items hand-labeled by sustainability experts; Tasks 4–5 required unanimous agreement on claims; Task 7 fields reviewed by ≥6 experts in a field-by-field pass.
  • Products span seven categories and 3–5 orders of magnitude in declared kgCO2e.

• Baselines and experimental protocol

  • Benchmarked models include GPT-5.5, GPT-5.4-mini, Claude Opus/Sonnet/Haiku, Gemini 3.1 Pro/Flash, DeepSeek v3.2, and Parakeet (mapping specialist).
  • Both single-shot and agentic/chained (compositional) inference evaluated; triage and mapping are separate agent tasks in the chain.
  • Primary aggregate acceptability threshold: within 2× of declared EPD kgCO2e (median |RE| and within-5× also reported).

Implications for AI Economics

• Measurement quality and economic signals

  • Corporate procurement, product comparisons, and consumer-facing labels depend on reliable product-level emissions. Systematic omissions or mis-mappings can distort relative carbon intensities across products, mispricing externalities and skewing demand signals that drive decarbonization investment.
  • Error cancellation (aggregate correctness with wrong internal allocations) undermines hotspot analyses and can misdirect capital toward ineffective emission-reduction levers.

• Market and regulatory risk

  • Many regulatory reporting regimes and voluntary carbon-related markets require auditable, traceable emissions accounting. Black-box aggregate estimators that fail to produce defensible intermediate outputs increase compliance risk and raise liability for firms relying on them.
  • Inaccurate PCFs can generate false compliance outcomes or misallocate subsidies/penalties tied to product footprints, with real economic transfers.

• Incentives for model design and procurement

  • Buyers and auditors will prefer systems that produce inspectable, compositional outputs with documented assumptions and evidence. Benchmarks like PCFBENCH show current LLMs need specialised modules (better retrieval, constrained numerical solvers, mass-balance checks, domain ontologies) to be economically useful.
  • There is value in hybrid systems: retrieval and mapping specialists (like Parakeet), constrained numerical modules, and human-in-the-loop validation for high-stakes reporting.

• Policy and research implications

  • Policymakers should be cautious about accepting black-box LLM estimates for disclosure or regulatory compliance; standards should require decomposable, auditable outputs and uncertainty reporting.
  • For researchers and practitioners, PCFBENCH provides targeted task-level evaluation to prioritize engineering: improve decomposition recall, robust ontology alignment, evidence-grounded extraction, and hard physical-consistency constraints (e.g., mass-balance enforcement).
  • Economic models of decarbonization that rely on ML-generated PCFs should account for systematic biases documented here (downward bias from omitted components; large-tail catastrophic mapping errors).

Suggestions for practitioners - Use compositional pipelines with strong retrieval/evidence components and enforce mass-balance checks rather than trusting single-call aggregate outputs. - Combine specialised mapping modules with LLMs for decomposition and extraction, and keep humans in loop for edge cases and regulatory submissions. - Adopt PCFBENCH (links above) to benchmark model updates and to quantify sources of error before deploying PCF automation in procurement, reporting, or market contexts.