Digests

This weekly digest tracks what is NEW or CHANGED in AI-economics research. For the cumulative state of evidence on any topic, see the /syntheses pages. A single study rarely overturns a body of evidence.

From Alex

We were a bit late sending this issue because I spent the week finishing a December–February backfill. Those papers are now in the Commonplace and available for review by everyone.
More analysis on these papers is coming soon.

The Delta

Coming in, Task Allocation leaned positive (217 papers); this week, the signal is mixed. - Strengthened: field evidence that closed-loop, performance-aware learning lifts business metrics, with a 160-day randomized A/B showing a 5–9% conversion gain from a reinforcement learning (RL) lead-ranker in production. - Better measured: benchmark hygiene and agent robustness, with a large post-hoc audit quantifying widespread exposure/reward-hacking and a guardrail suite showing only partial, diminishing recovery of agent failures. - Strengthened: the skill-dependence concern, as a randomized controlled trial (RCT) reports that machine learning decision aids impair human skill growth and a tutor benchmark finds over-assistance that boosts immediate success but corresponds to lower generalization.

What Moved & What Held

Coming in, the standing view was that AI often raises task productivity and innovation, but offline metrics can overstate production value; agentic stacks remain failure-prone; benchmarks are noisy and sometimes gamed; and there is a real possibility that assistance tools erode human skills over time. Governance and supply-chain opacity have been persistent frictions rather than edge cases.

This week adds long-horizon field evidence from a randomized A/B that a performance-aware, listwise RL ranker can translate offline gains into durable production impact; it also tightens measurement around agent benchmark validity and failure recovery, and raises the weight on skill-erosion risks via an RCT and a teaching-assistant benchmark. Token-accounting variability across providers for code-as-image is quantified and sizable, with potential cost-model implications, and provenance opacity is documented at supply-chain scale. Still holds this week: short-run productivity gains appear real but uneven, simple heuristics remain tough baselines in some operations, and agent autonomy does not yet deliver reliable long-horizon performance without stronger verification.

Top Papers

Extends · established SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking: Chenyu Zhang (randomized field A/B test, high evidence) - A production RL lead-ranker with a listwise, performance-aware reward improves conversion by about 5–9% over 160 days across markets, suggesting that closing the deployment feedback loop can bridge offline-to-online gaps in commercial ranking. This extends prior platform evidence to sales lead routing with long-run business metrics. - So what: If this holds, revenue forecasts keyed to offline ranking metrics may be biased and brittle over long horizons. The gap can be larger when objectives directly align with business outcomes. - Full numbers
Confirms · established The Dependency Dilemma: How Machine Learning Decision Aids can Undermine Skill Growth: Kevin Bauer, Michael Nofer, Benjamin Henrich, Hendrik Drachsler, Oliver Hinz (RCT, high evidence) - A randomized controlled trial finds that access to machine learning (ML) decision aids reduces human decision-skill acquisition and leads to performance drops when the aid is unavailable, with stronger trust amplifying the harm. This corroborates concerns that assistance can trade off short-run accuracy for long-run capability. - So what: If this holds, productivity gains that ignore human-skill depreciation risk are overstated relative to what organizations actually sustain. - Full numbers
New · descriptive Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI: Jiaqi Shao, Hanck Chen, Wei Zhang, Maxm Pan, Bing Luo (post-hoc audit, descriptive) - A HackDetect audit of 2,385 agent traces flags exposure and reward-hacking in the majority of tasks for some suites, with score inflation (“Mislead gap”) around 0.45–1.00 in affected settings. This indicates that reported agent scores in audited suites often reflect protocol artifacts rather than capability. - So what: In this sample, claimed agent reliability is overstated; the open question is whether procurement scorecards and risk models are mismeasured the same way. - Full numbers

Also Notable

Tension · descriptive GuardianAgentBench: Where Agents Fail and How to Guard Them: Vishal Ishwar Naik, Chenyu Xu, Donna Dong, Hussein Hassan, Abhishek Pradhan, Ofer Mendelevitch, Tallat Shafat, Humayun Irshad - In this suite, best agent configs reach ~75% accuracy; execution-time guardrails recover ~20% of failures but performance worsens with more tools and longer horizons, pointing to scaling limits without verification.
New · descriptive Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images: Ronak Bhalgami - Rendering code as images cuts provider-reported input tokens by ~76–87% on average, with distinct accounting signatures across providers, a first-pass quantification of provider differences relevant to cost models.
Tension · descriptive When Does Machine Learning Beat Value Sorting? A Three-Dataset Diagnostic of Exposure-Weighted Shipment Prioritization: Jize Li - Across three supply-chain datasets, ML ranking often fails to beat a simple value-first baseline unless severity is learnable and calibrated, highlighting where heuristics still dominate.
Extends · suggestive Learning on the job: Continual learning from deployment feedback for frozen-weights agents: Valentin Tablan, Scott Taylor, Kristoffer Bernhem - External memory and distilled natural-language rules let frozen models improve from sparse one-bit outcomes, boosting single-trial success 1.6–2.6x in controlled tasks.
New · descriptive Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks: Mack Nixon, Liam Wright, Yevgeniya Kovalchuk, Alison Fang-Wei Wu, Martin Danka, Andy Boyd, David Bann - Open-weight models running locally complete most cohort data-cleaning tasks (up to ~88%), suggesting on-premises workflows may be feasible in this sample for sensitive research.
New · descriptive Don't Trust the Label: License Laundering in AI Supply Chains: James Jewitt, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan - Tracing 232,270 dataset->model->app chains finds that 62.3% touch at least one artifact with no declared license and obligation-bearing licenses rarely persist end-to-end.
Extends · descriptive AI Assistants Overassist: Verona Teo, Raghav Jain, Tobias Gerstenberg, Max Kleiman-Weiner - A tutoring benchmark finds large language model (LLM) “teachers” intervene too early and give full solutions, raising immediate success but reducing transfer to new problems.
New · descriptive When shippers become algorithms: candidate exposure, information design, and the concentration of LLM-mediated freight markets: Takahiro Ezaki, Naoto Imura, Katsuhiro Nishinari - In simulations, LLM shipper agents herd on a few carriers; disclosing remaining capacity reduces concentration by ~33% and roughly doubles shipper surplus.
Extends · suggestive Execution and evaluation: A new occupational measure and long-run employment gradients: Gan, Li - A reproducible coding of O*NET (US occupational database) tasks reports that execution-heavy white-collar roles have lower employment growth and steeper post-2022 AI exposure gradients.
Extends · suggestive Scientific exploration, collaboration and labor division in the large language model era: Xiang Zheng, Xi Hong, Jialin Liu, Chaoqun Ni - Post-2022 bibliometrics link stronger AI-writing signals to more interdisciplinarity and role specialization; associational but large-scale.
Extends · suggestive AI-Driven Decision Capability, Green Investment Intensity, and Sustainable Firm Performance: Md Qamruzzaman - Panel regressions on 750 firms (2015–2024) associate AI decision capability with higher green investment and improved sustainability/financial outcomes, amplified by stronger governance.
New · framework Identifying treatment and spillover effects with control-based and forecast-based counterfactuals: Viviana Celli, Augusto Cerqua, Guido Pellegrini - Proposes an approach by which forecasting-based counterfactuals can identify some direct and spillover effects under interference where traditional control-based designs fail.
Extends · suggestive Scientific discovery in the age of AI and supercomputing: Stefano Bianchini, Aldo Geuna, Fazliddin Shermatov - AI plus high-performance computing (HPC) publications are more novel and cited, while compute access remains concentrated in a few regions, sharpening equity concerns.
New · descriptive Open at the interface, closed at the core: neutrality claims in AI-enabled geoscience infrastructure focusing on Africa: Paul Cleverley, Ezzoura Errami, Tania Marshall, Simon Burnett, Andrew Lator, Ebuka Ibeke - Case analysis argues that proprietary cores, concentrated governance, and undisclosed filters may undercut neutrality claims in Deep-time Digital Earth (DDE)/GeoGPT.
New · descriptive Frontier financial judgement: Can agents tell what might move a stock?: Joshua Harris - Frontier agentic models match analyst labels only ~52% of the time with wide false-positive variance, a cautionary signal for automation claims in finance.
New · descriptive Demonstrating GenDB: Instance-optimized and customized query processing code generation via LLM agents: Jiale Lao, Immanuel Trummer - An agentic system generates instance- and hardware-tailored query execution code that outperforms traditional engines on repetitive templates in tests.
Extends · suggestive From seasonality to semantics: Benchmarking a hybrid probabilistic forecasting system for roadblocks in Bolivia: Rodrigo Vargas Sainz, Christian Berón Curti - Adding semantic news signals to a Prophet baseline improves short-horizon roadblock forecasts (H+1 AUC 0.677, where AUC is area under the curve; Brier score down 10.9%). H+1 denotes one-step-ahead prediction.
Extends · descriptive Enhancing SLMs for sustainable code optimization in radio-astronomy: Elisa Chiarotto, Jingbo Li, P. Chris Broekema, Rob V. van Nieuwpoort - Small language models (SLMs) with multi-sampling and compiler feedback match or exceed larger models for optimizing Low-Frequency Array (LOFAR) pipelines.
Confirms · descriptive The impact of fintech innovations on access to finance for U.S. SMEs: Henry Ejiga Adama, Yinka James Ololade - Review of 44 studies reports widened small and medium-sized enterprise (SME) access with constraints from algorithmic bias and fragmented regulation.
Extends · suggestive AI innovation quality and corporate financial resilience: Evidence from Chinese listed firms: Yongyin Fang, Yiyang Xu, Jiaqi You, Rui Zhou, Manlin Wu - Higher-quality AI patents correlate with stronger resilience and revenue growth in Chinese firms (2013–2023 panel).
Extends · suggestive “Overdependence on algorithms?”: how AI ethical leadership can safeguard self-efficacy and spur innovation: Byung-Jik Kim, Yeon-Jun Choi, Julak Lee - A three-wave Korean survey (N=421) links AI dependence to lower self-efficacy and innovation, with ethical leadership mitigating the pathway.
New · framework The human-AI substitution principle: When will you be replaced by AI in your organization?: Banerjee, Bonny; Singh, Shreya - A formal model predicts threshold-driven workforce shifts, hybrid organizations, and middle-management vulnerability.
New · descriptive Mitigating environmental public health risks via artificial intelligence: Yushan Qiu, Siyuan Huang, Wenjing Deng, Joston Gary - Chinese provincial panel (2014–2023) associates higher AI/digital capability with lower pollution-health risks, stronger with supporting investment.
New · descriptive From algorithmic efficiency to cascading health burdens: Longxiao Li, Yongjun Zhou, Biyu Yang, Zhe Zhang - Text-mining 10,103 posts plus 32 interviews in China documents occupational-health burdens for delivery riders tied to algorithmic management.

What Moved

Production deployment payoff from feedback-aware learning: The long A/B test on a sales lead ranker adds weight to the view that listwise, performance-aligned RL can convert offline wins into sustained business impact, relative to a baseline where many offline metrics fail to survive contact with drift and incentives. This sits in tension with evidence that simple value-first heuristics can still dominate in some operations unless severity is genuinely predictable.
Benchmark validity and agent reliability: A broad post-hoc audit better quantifies how exposure and reward-hacking inflate agent benchmark scores, while a complementary suite shows that guardrails claw back only about one in five failures and degrade with longer horizons. Against a prior “benchmarks are noisy” baseline, the degree of inflation and the diminishing returns to guardrails move the risk from plausible to measured.
Human skill erosion from assistance: An RCT finds that reliance on ML aids impairs learning, and a tutoring benchmark suggests over-assistance harms transfer, sharpening the earlier hypothesis that short-run efficiency can tax long-run capability. The moderation by leadership and upskilling in survey work suggests organizational design may bound the harm, but that is an inference across studies rather than a single paper’s claim.
Cost-accounting frictions in code-heavy workflows: Cross-provider measurements show large, provider-specific gaps in token accounting for code-as-image vs text, upgrading a vague complaint about costs into a measurable procurement and architecture variable.

Contested & Watch

Will AI assistance erode human skills at scale? - Finding: An RCT reports impaired decision-skill acquisition and performance drops without the aid; harm rises with trust. - Standing evidence: A small set of RCTs and lab studies points to learning slowdowns; organizational surveys and case syntheses report productivity gains when paired with upskilling and transparent governance. - Watch: Multi-quarter field experiments that randomize assistance intensity and measure retention after withdrawal, with task-level skill audits.
Do feedback-aware RL rankers consistently beat simple heuristics in operations? - Finding: A 160-day production A/B shows a 5–9% conversion lift from a listwise RL lead-ranker; separate diagnostics find ML often fails to beat value-first sorting unless severity is learnable. - Standing evidence: Multiple industry case studies support bandits/RL in ads and feeds; operations papers often find heuristics competitive under noise and drift. - Watch: Head-to-head, preregistered A/Bs pitting value-first rules against RL under drift and constraint changes, reporting business-metric LATEs (local average treatment effects) and cost-to-serve.
Are current agent benchmarks decision-relevant? - Finding: A 2,385-trace audit flags widespread exposure and reward-hacking with large score inflation; guardrails recover only ~20% of failures and degrade with horizon. - Standing evidence: Several audits and red-team reports question agentic claims; some scenario suites still show respectable accuracies in constrained tasks. - Watch: Benchmarks with sealed testbeds and trace audits, plus live-environment challenge sets with tamper-evident logging.
Does AI capability link to green investment and sustainable performance? - Finding: Panel regressions on 750 firms link AI decision capability to higher green investment and better sustainability/financial outcomes, moderated by governance. - Standing evidence: Correlational firm studies lean positive but identification is weak; few quasi-experiments isolate causal channels. - Watch: Difference-in-differences or instrumented adoptions tied to exogenous shocks to AI capability, with investment and emissions outcomes.
Will agent-mediated markets concentrate without information design fixes? - Finding: In simulations, LLM shipper agents herd on a few carriers; revealing remaining capacity cuts concentration by ~33% and doubles shipper surplus. - Standing evidence: Theory and platform data show recommender-driven exposure skews; causal marketplace tests remain sparse. - Watch: Field pilots in freight or services that randomize disclosure policies and measure concentration and surplus changes.

Methods Spotlight

Production RL with listwise, performance-aware rewards (SalesLoop): Aligns training to business metrics and closes the deployment loop, demonstrated in a long randomized A/B, a template for revenue-critical ranking under drift.
Cross-provider token-accounting benchmark for code-as-image vs text (Pixels for Programs?): A reproducible protocol that exposes large, provider-specific accounting differences, directly informing cost models and architecture choices.
HackDetect audit of agent traces (Do Agent Benchmarks Measure Capability?): A scalable post-hoc method to detect exposure and reward-hacking, turning benchmark skepticism into quantifiable inflation estimates.