Pooling interaction histories to create generic agent 'skills' improves coding-agent task completion more reliably than developer-specific personalization; personalization helps only when preferences recur frequently across similar tasks.

Main Finding

Personalized, developer-specific skills distilled from limited developer–agent interaction histories provide only small and inconsistent improvements to coding-agent task performance. In contrast, a generic skill distilled by pooling interaction traces across developers yields more consistent and larger gains.

Key Points

• Research question: Can natural-language, developer-specific skills (SKILL.md) distilled from past developer–agent interactions generalize to future tasks and reduce developer pushback?
• Skill generation is two-stage:
  • Bootstrap: pattern-analysis template to extract task-independent candidate rules (communication, work style, follow-up handling, validation/commit preferences).
  • Evidence-grounded refinement: LLM refiner keeps/revises/adds/removes rules only if supported by ≥2 user turns from different sessions.
• Evaluation uses an LLM-based trajectory-conditioned developer simulator (replay), so the agent may take different trajectories under different skill conditions while the simulator issues follow-ups based on the evolving trajectory.
• Experimental conditions compared: (A) no skill, (B) personalized skill (from same developer), (C) random other-developer skill, (D) generic pooled skill (includes all developers).
• Key quantitative results (mean task-completion score ±SE):
  • No skill: 65.02 ± 3.24
  • Personalized: 65.99 ± 2.14 (+0.97)
  • Random developer skill: 65.94 ± 3.66 (+0.92)
  • Generic pooled skill: 68.80 ± 2.26 (+3.78)
• Follow-up rates (percentage of replay runs where simulator issued ≥1 non-terminal follow-up):
  • No skill: 24.76%
  • Personalized: 30.95%
  • Random: 27.62%
  • Generic: 30.00%
• Win/Tie/Loss vs no-skill (percentages):
  • Personalized: 41.43 / 14.76 / 43.81
  • Random: 43.33 / 17.62 / 39.05
  • Generic: 50.95 / 14.76 / 34.29
• Statistical notes: improvements for personalized skill were not significant (paired t-test p = .399); generic improvement over no-skill did not reach conventional significance (p = .063) but was the most consistent across splits.
• Further analysis: personalization becomes more effective when developer preferences recur frequently and when the developer works on similar tasks over time (i.e., sufficient, consistent signals make developer-specific guidance generalizable).

Data & Methods

• Dataset: Filtered subset of SWE-chat (Entire.io) — 206 replayable CLI coding-agent sessions from 13 developers (Jan–Jun 2026) after strict filtering for reproducible runs. For each developer: 80% evolution (164 sessions total) and 20% held-out test (42 sessions).
• Tools and models:
  • Codex 1 with GPT-5.5 for: skill generation (bootstrap + refiner), coding-agent execution (Codex CLI), developer simulator, and task-completion scoring (LLM-as-judge).
• Replay setup:
  • For each held-out session, construct a task summary from developer-authored messages (simulator gets the summary; agent gets original first-turn request).
  • Simulator inspects agent responses and issues focused follow-ups or stops; each replay limited to ≤6 developer–agent turns.
  • Skill inserted into agent prompt at start; skill unchanged during the session.
• Evaluation metrics: task-completion score (SWE-chat 100-point rubric judged by LLM), follow-up rate, win/tie/loss in paired comparisons. Experiments repeated over five random seeds for within-developer splits.
• Constraints and potential biases:
  • Small sample of developers (13) and sessions (206) due to strict reproducibility filtering.
  • Generic skill was pooled across all developers and therefore included the target developer’s data (benefit from pooling may partly reflect that inclusion).
  • Use of LLM-based simulator and LLM-as-judge introduces model-based evaluation biases.
  • Limited interaction depth (max 6 turns) and reliance on Codex/GPT-5.5 may influence generality.

Implications for AI Economics

• Returns to pooling data (economies of scale): Pooling interaction traces across users produces broadly useful procedural guidance that outperforms limited per-user personalization. From a product/economic perspective, investing in centralized, shared skill distillation can yield higher expected performance benefits per unit data than attempting lightweight per-user personalization when user histories are sparse.
• Cost-benefit and targeting of personalization:
  • Personalization has value only when users generate enough repeatable, consistent signals (heavy users, teams with homogeneous tasks). Firms should target personalization investments where interaction volume and task homogeneity are high to overcome data sparsity.
  • For the majority of users with few or heterogeneous interactions, generic pooled skills offer better ROI.
• Privacy and data governance trade-offs:
  • Pooling across developers is beneficial but can raise privacy/legal concerns (proprietary codebases, sensitive patterns). Product design should consider privacy-preserving aggregation or opt-in models for pooling that capture the economic upside while respecting constraints.
• Product and monetization strategies:
  • Build robust, shared “skill libraries” (procedural guidance) as core, low-cost upgrades that benefit many users.
  • Offer premium, targeted personalization for power users or teams with sufficient interaction history (a tiered monetization model).
  • Lightweight, prompt-based skills are a low-cost mechanism (no model fine-tuning) to capture and deploy experience—this reduces infrastructure and update costs versus retraining models.
• Implications for competition and platform value:
  • Platforms that can aggregate large, cross-user interaction datasets may obtain durable advantages by producing higher-quality generic skills; network effects from more interactions increase the value of pooled skills.
• Research and operational recommendations:
  • Prioritize collection of richer, longitudinal interaction data for users where personalization is expected to pay off.
  • Improve evidence thresholds and methods (e.g., better distinguishing one-offs from stable preferences) and explore hybrid approaches: start with generic skills, then adapt when sufficient user-specific evidence accumulates.
  • Validate findings with human-in-the-loop studies and diverse model families to reduce evaluation biases from LLM-based simulators and judges.

Overall takeaway: invest first in broadly transferable procedural guidance derived from pooled interactions; selectively deploy developer-specific personalization only where repeatable, consistent, and task-similar signals justify the additional cost.