Step-level credit signals used to train LLM agents fail to find the actions that actually change outcomes in a replayable tool environment; scores mostly echo model fluency, not causal effect, and apparent training differences reflect dose rather than meaningful credit.

Main Finding

Audited against causal ground truth produced by executed replay in a replayable single‑agent tool environment (ALFWorld), common step‑level credit signals used to train LLM agents — (1) the implicit outcome‑conditioned log‑probability ratio used by HCAPO, (2) LLM‑judge scores, and (3) the policy’s own confidence — do not identify which steps causally affect outcomes any better than a marginal‑matched shuffled control. Instead, these credit signals mostly echo the policy’s fluency (action probability), and apparent differences between credit rules in training are explained by effective sample size (“dose”), not by the credit content.

Key Points

• Ground truth construction: executed replay. At each decision point they resample K=4 admissible alternatives from the same policy, roll each alternative to termination, and measure Areplay(t) = mean(outcome | factual replays) − mean(outcome | alternative rollouts).
• Causal contribution is sparse:
  • Qwen2.5‑7B: 30.5% of complete turns are pivotal (measurable Areplay ≠ 0).
  • 69.5% of complete turns have Areplay = 0 at the study’s resolution (i.e., indistinguishable from factual within sampling bounds).
• Measurability is model‑dependent:
  • Policy‑supported counterfactuals undefined at 13.1% of intervened turns for Qwen vs 26.8% for Llama‑3.1‑8B (a 2.05× difference).
  • Where counterfactuals are defined, Llama had a higher pivot rate (38.3% vs Qwen’s 30.5%).
  • Excluded turns are systematically low‑entropy (high fluency); for Qwen the included–excluded mean policy log‑prob diff = −0.70.
• Fidelity audit results:
  • Within‑trajectory Spearman correlation between credit and Areplay (median across trajectories) is indistinguishable from a marginal‑matched shuffled control for each credit family (verdict class H3, placebo‑level).
  • Reported examples: Qwen implicit ρ̂ = 0.0193 [−0.109, 0.081] vs shuffle [0.005, 0.114]; Qwen judge 0.1142 [0.027, 0.168] vs shuffle [−0.049, 0.117]; Llama implicit −0.043 [−0.125, −0.016] vs shuffle [−0.102, +0.024] (control comparison binds verdicts).
  • Per‑step sign agreement and precision‑at‑pivotal also show no reliable causal identification (implicit family mildly anti‑concentrated; judge family shows some sign agreement but no concentration on pivotal turns).
• Mechanism: credit largely echoes fluency
  • Median rank correlation between implicit credit and policy log‑probability ≈ +0.75 (replicated across families).
  • In joint regressions (credit ~ fluency + causal increment), fluency accounts for most variance; partial correlation between credit and Areplay after conditioning out fluency ≈ −0.004 (p≈0.87 for Qwen).
• Practical decision rule (cost‑only): a frozen confidence router can cut judge calls by ~13.1% per turn (14.0% per trajectory) while recovering pivotal steps only at chance — a tradeoff in cost without improved causal selection.
• Training experiment: seven‑arm pre‑registered training runs show no arm reliably improves upon the untrained policy. Apparent checkpoint differences are explained by training dose (sparser credit → fewer retained examples → far fewer optimizer steps), so comparisons must dose‑match or they measure dose, not credit.
• Protocol contributions: the paper proposes standards for credit comparisons (dose matching, measured perturbation strength for controls, MDE ladder, integrity taxonomy).

Data & Methods

• Environment and policies:
  • ALFWorld single‑agent tool environment (deterministic prefix‑restore verified).
  • Policies: Qwen2.5‑7B‑Instruct (primary), Llama‑3.1‑8B‑Instruct (replication; one corrected re‑run after a template defect).
  • Collection: trajectories sampled with temperature 0.7 using the HCAPO agent template.
• Executed replay instrument:
  • At each decision turn t: re‑execute factual action ≥3 times; sample K=4 distinct admissible alternatives (within ≤300 seeded draws) and roll each alternative ≥3 times to terminal. If fewer than 4 distinct alternatives can be sampled, that turn is excluded and counted (i.e., undefined policy‑supported counterfactual).
  • Areplay(t) computed as the difference in mean outcomes between factual replays and alternative rollouts.
  • Per‑turn noise floor σfloor(t) from factual replica variance; zeros are explicitly resolution‑bounded (no significance test applied for pivot classification).
• Metrics and inference:
  • Primary fidelity metric: within‑trajectory Spearman correlation between credit scores and Areplay, aggregated as median across trajectories with 10,000‑resample bootstrap CIs.
  • Controls: marginal‑matched within‑trajectory shuffles (preserve each trajectory’s marginal distribution) and uniform/random baselines.
  • Additional checks: per‑step sign agreement vs chance, precision‑at‑pivotal lift, partial correlations conditioning out fluency, three structural bias tests with Holm correction, frozen verdict order (controls gate before effect gate), pre‑registered exclusion rules.
  • Replay budget and coverage: Qwen run had 2,034 intervened turns (1,768 complete after exclusions); Llama run had coverage ≈88.3% after correction (792 complete turns reported for pivotal stats).
• Reproducibility: thresholds and gates pre‑registered and frozen; artifacts regenerated by script; cross‑family replication criterion pre‑registered and met.

Implications for AI Economics

• Causal vs correlational credit: common step‑level credit signals used in training pipelines appear to reflect fluency and selection bias rather than causal contribution. Economically, investments expecting improved sample efficiency or faster learning from these credit signals should be re‑evaluated until causal benefit is demonstrated.
• Measurement and evaluation: policymakers, researchers, and procurement teams should require causal audits (e.g., executed‑replay or equivalent counterfactual methods) rather than correctness/annotation benchmarks when the goal is to buy or certify credit rules that meaningfully improve outcomes.
• Cost vs benefit of judge infrastructure: deploying expensive LLM judges or complex outcome‑conditioning methods may not yield causal gains despite improving surface evaluation metrics. Cost‑saving hacks like confidence routing can reduce judge calls but do not meaningfully recover pivotal steps; cost reductions should be balanced against the absence of confirmed efficacy.
• Selection and “dose” confounding: reported benefits of credit rules in training can be driven by effective sample size (dose) differences — e.g., sparser credit retains fewer examples and thus changes optimizer exposure. Economists and experiment designers must control for effective sample size when comparing training rules, or results will misattribute gains to method content rather than differing optimization exposure.
• Market and R&D allocation: funders and firms should be cautious allocating large R&D budgets to scale up judge‑based or implicit credit mechanisms without causal evidence of impact. Redirected resources may be better spent on methods that demonstrably affect downstream outcomes or on building auditing capabilities.
• Auditability and regulation: measurability of causal contribution depends on the model and is not an environment constant; regulatory compliance frameworks that rely on audits must account for policy‑dependent measurability, sampling budgets, and selection biases (e.g., low‑entropy actions that lack counterfactuals).
• Research priorities and risk management:
  • Require dose‑matched, causal evaluations before claims of step‑level credit efficacy.
  • Invest in broader causal audit tooling (multi‑agent, stochastic environments, larger models) to test generality.
  • Consider economic analyses of the ROI of judge infrastructure vs simpler baselines given null causal gains.
  • Monitor potential misallocation of compute/training budgets driven by misinterpreted credit benchmarks.
• Limitations to note for economic interpretation:
  • Single environment (ALFWorld) and relatively small models in this study; results may not fully generalize to other environments (e.g., real‑world tool stacks), larger models, or multi‑agent settings.
  • Executed replay is constrained by sampling budgets (K, rollouts) and model entropy; some causal effects may be below the study’s resolution.

Overall, the paper cautions that widely adopted step‑level credit signals may not produce the causal information training loops presume, and it provides a reproducible audit protocol that AI economists, procurement agents, and regulators can use to vet claims about the causal value of credit rules before large investments or deployments.