Procedural memory measurably improves LLM agents on enterprise tasks: a single refinement raises aggregate accuracy by 3.7–6.7 points, and skills learned from diverse multi-model traces transfer across models with 73.1% test accuracy; some skills generalize broadly while others become role-specific and lose transferability.
Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation
AFTER shows that procedural memory improves LLM agent performance on enterprise workflows—single-round refinement yields 3.7–6.7 point aggregate gains, and skills distilled from diverse multi-model traces achieve substantially higher cross-model transfer (73.1% accuracy) than single-model sources.
Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. We introduce AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills, designed to evaluate how skills transfer across tasks, roles, and model backbones. The benchmark includes controlled evaluation settings for local improvement, cross-task transfer, cross-role transfer, and cross-model generalization. Experiments show that procedural memory delivers consistent gains in industrial workflows: a single refinement round improves aggregate performance by 3.7-6.7 points, while skills evolved from diverse multi-model execution traces achieve 73.1% cross-model test accuracy, outperforming all single-model trace sources. We further find that some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer. These results provide practical guidance for building, evaluating, and deploying procedural memory systems in production agent platforms.
Summary
Main Finding
Procedural memory—compact, versioned skill artifacts used by LLM agents—improves workplace-task performance and inference efficiency, but its economic value depends strongly on how skills are produced. Skills evolved from diverse, multi-model execution traces generalize much better (73.1% cross-model test accuracy) than those derived from single-model or narrow experience, while skills trained on narrow traces often over-specialize to source workflows and lose transferability across roles and models.
Key Points
- AFTER benchmark: 382 realistic enterprise tasks, 6 professional roles (DE, DS, GenAI, Infra, PM, SWE), 22 procedural skills; includes controlled splits for in-role, cross-task, cross-role, and cross-model transfer.
- Two accuracy metrics:
- M1: partial progress (fraction of tests passed)
- M2: full-pass task success
- Static skill value:
- Adding procedural skills (handcrafted or LLM-generated) yields average full-pass improvements of +2.8 percentage points.
- LLM-generated skills often match or beat handcrafted skills.
- Single-round refinement:
- One LLM-guided refinement pass (on handcrafted skills) yields an additional +5.2 points on average; larger models gain more.
- Skill evolution and transfer:
- Skills evolved from diverse, multi-model traces achieved 73.1% cross-model test accuracy, outperforming best single-model trace sources by at least +13.7 points.
- Narrow experience often produces high in-context gains but reduces generality (overfitting to source workflow); diverse experience yields both high specificity and generality.
- Cross-role transfer is limited:
- Skills optimized for one role can hurt performance when applied to another (example: pdf skill gives +11.7 M2 within PM but −7.5 when transferred to DS).
- Efficiency gains:
- Evolved skills reduce inference token usage substantially (example: −62% tokens for Claude, −16% for Hermes on a Kafka task), indicating direct cost savings.
- EVOLUTION harness and SKILL.md:
- Procedural memory is managed as versioned SKILL.md artifacts; EVOLUTION standardizes trace collection, update (COLLECT–DIAGNOSE–REVISE–PROMOTE), lineage, and evaluation.
- Framework comparison:
- Multiple evolution frameworks (Hermes, Codex, Memento, MemP, EvoSkill) differ substantially: training gains do not always imply held-out transfer gains.
Data & Methods
- Benchmark composition:
- 382 tasks: 318 single-skill, 64 multi-skill workflows; 56 adapted from prior benchmarks, 326 newly authored or expert-refined.
- 22 skills across document processing, data ops, ML/AI, infrastructure, and software engineering.
- Skills provided in two prompt bodies: handcrafted (H) and LLM-generated (G).
- Evaluation protocol:
- Controlled splits for measuring specificity (ptgt = psrc) and generality (shifted tasks, roles, or models).
- Traces collected from agent executions tied to skill versions; update operator U maps traces to new skill versions (Σ′ = U(Σ, D)).
- Reflector ρ (e.g., an LLM) inspects traces to propose revisions; EVOLUTION enforces reproducibility and versioning.
- Experiments:
- Static evaluation across many LLMs (various sizes and vendors) using no-skill, H, and G conditions.
- Single-round LLM-guided refinement (Codex used as reflector in reported experiment).
- Multi-round evolution with different frameworks and trace budgets (narrow: n=1 trace source; diverse: n=5 sources).
- Cross-model and cross-role transfer experiments; token-usage measurement to quantify inference cost.
- Key quantitative results:
- Static M2 uplift: average +2.8 points from skills; single refinement +5.2 points.
- Cross-model best: 73.1% test accuracy from diverse multi-model traces vs. 36.0–59.4% for single-model sources.
- Token reductions example: −326k tokens (62%) for one model; −48k tokens (16%) for another.
Implications for AI Economics
- Direct inference-cost savings
- Evolved procedural skills can materially reduce token consumption per workflow (examples up to −62%), translating to lower per-query inference costs—valuable for high-volume enterprise deployments.
- Better ROI from modest models + memory
- Weaker or mid-tier models often benefit more from injected procedural memory than frontier models, and mixed-source (including weaker-model) traces produced the most transferable skills. This suggests a cost-effective strategy: invest in building procedural memory and use smaller, cheaper runtime models, reducing licensing and compute spend.
- Data-collection and labeling trade-offs
- Diverse trace collection (multi-model, multi-role) is more valuable than large quantities of homogeneous traces. Economically, resources should prioritize diversity in logging and instrumentation over sheer volume.
- Model upgrade and vendor lock-in strategy
- Because procedural skills can generalize across models when evolved from diverse traces, firms can reduce vendor lock-in risk: investing in transferable SKILL.md artifacts eases backbone model swaps, lowering migration costs.
- Maintenance and operating costs
- Procedural memory requires infrastructure (EVOLUTION-like tooling), human-in-the-loop validation, and periodic re-evolution. These fixed costs must be balanced against per-query savings; single-round refinement yields meaningful gains, so even modest maintenance can be cost-effective.
- Risk of brittle specialization
- Over-specialization to roles or workflows can degrade performance when contexts shift, introducing operational risk and potential hidden costs (rework, failures). Firms should monitor cross-role performance and maintain evaluation splits that reflect realistic distribution shifts.
- Product strategy and pricing
- Vendors can monetize skill libraries, versioning, and evolution services (e.g., "skill vaults" or managed evolution pipelines). Pricing models might combine fixed fees for skill development and usage-based fees tied to inference savings or task success metrics.
- Procurement and benchmarking
- Procurement decisions should include assessments of procedural-memory capabilities and transfer metrics (not just base-model accuracy). Benchmarks like AFTER provide a framework to value skill-transfer robustness in vendor comparisons.
- Labor substitution and productivity
- Procedural skills that reliably automate recurring workplace tasks can increase agent productivity and potentially substitute for labor in repetitive workflows—this has economic implications for labor allocation, skill training, and organizational design.
Limitations to consider when applying these implications: AFTER focuses on tech-sector procedural tasks, evaluation uses automated test suites (functional correctness) rather than human-centered metrics, and the experiments capped trace budgets—real-world returns may vary with larger-scale trace accumulation and domain differences.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents systematic, controlled benchmark experiments across 382 enterprise tasks and multiple LLM backbones, showing consistent performance gains and cross-model transfer; however, evidence is limited to benchmark settings (no field/real-world productivity measures), model/sample selection may bias results, and causal claims about workplace productivity are not established.
Methods Rigormedium — Design includes controlled evaluation settings (local improvement, cross-task/role/model transfer) and comparisons of single-model vs multi-model traces, which is methodologically sound for benchmarking, but the rigor depends on task representativeness, evaluation metrics, statistical testing details, and potential selection/engineering choices that are not validated in live deployments.
SampleAFTER benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills; evaluated under controlled settings for local refinement, cross-task transfer, cross-role transfer, and cross-model generalization using execution traces from single-model and multi-model sources across multiple LLM backbones.
Themesproductivity human_ai_collab
GeneralizabilityTasks limited to the 382 benchmark items and six roles—may not cover full enterprise workflow diversity, Evaluations run on a limited set of model backbones; results may not hold for substantially different architectures or scales, Benchmarked accuracy/performance metrics may not directly translate to real-world productivity or economic outcomes, Multi-model trace sources and data collection practices may reflect engineering choices not generalizable across organizations, Potential language, domain, and cultural biases if tasks skew toward particular industries or geographies
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| We introduce AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles and 22 procedural skills. Other | null_result | benchmark size and coverage (number of tasks, roles, skills) |
Reading fidelity
high
Study strength
high
|
n=382
|
| The benchmark includes controlled evaluation settings for local improvement, cross-task transfer, cross-role transfer, and cross-model generalization. Other | null_result | availability of controlled evaluation settings |
Reading fidelity
high
Study strength
high
|
n=382
|
| Procedural memory delivers consistent gains in industrial workflows: a single refinement round improves aggregate performance by 3.7-6.7 points. Output Quality | positive | aggregate performance (task performance) after procedural memory refinement |
Reading fidelity
high
Study strength
medium
|
n=382
3.7-6.7 points
|
| Skills evolved from diverse multi-model execution traces achieve 73.1% cross-model test accuracy, outperforming all single-model trace sources. Output Quality | positive | cross-model test accuracy of evolved skills |
Reading fidelity
high
Study strength
medium
|
n=382
73.1% cross-model test accuracy
|
| Multi-model execution-trace-derived skills outperform skills derived from any single-model trace source. Output Quality | positive | relative performance (accuracy) of skills from multi-model traces versus single-model traces |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Some skills generalize broadly across tasks and models, whereas others become specialized to role-specific workflows and lose effectiveness under transfer. Skill Acquisition | mixed | skill transfer effectiveness (generalization versus specialization under transfer) |
Reading fidelity
high
Study strength
medium
|
n=22
|
| Procedural memory is increasingly used to improve LLM agents on recurring workplace tasks, yet its ability to produce reusable skills remains poorly understood. Other | null_result | state of understanding of procedural memory's ability to produce reusable skills |
Reading fidelity
medium
Study strength
speculative
|
not reported
|