Short sequences of human–agent interaction materially raise agent performance: in experiments with 30 expert users on writing and visual-creation tasks, test-time context and weight adaptation lift agent solo success by about 4.5–20.9% within tens of sessions. An evolving verifier that crystallizes user criteria also flags 16–22% more failures than rubrics from LMs or humans alone, and some personalized behaviors generalize across users.

Main Finding

Test-time adaptation through human-agent interaction (TAHI) — which ingests multi-session human feedback (plans, edits, messages, and evolving rubrics) into both agent context and weights — enables rapid, efficient personalization of LLM-based agents. Across 30 experts and 600 tasks in writing (paper abstracts) and visual creation (HTML figures), personalized agents improved first-pass (solo) success by roughly 4.5–20.9% within only tens of sessions; the system’s evolving rubrics also detect 16.0–22.3% more failures than rubrics produced by LMs or humans alone.

Key Points

• TAHI: a streaming, test-time adaptation framework where agents update after each completed human-agent session to better match an individual user’s unspoken preferences and procedures.
• Interaction channels: the interface supports Plan (structured task decompositions), Deliverables (direct file/visual edits), Message (text feedback), and Rubrics (verifier criteria) so humans can give rich, multi-modal signals.
• Two adaptation modes:
  • Context-based adaptation: induce and store user-specific memory (preferences/facts) and skill library (procedures) as editable context (verbalized by an LM).
  • Weight-based adaptation: train LoRA adapters via Direct Preference Optimization (DPO) using preference pairs derived from the agent’s initial vs. final (human-refined) solutions, consolidated into one-shot supervision trajectories.
• Evolving verifier/rubric module: an LLM crystallizes human interactions into explicit, gradable rubric items which users can correct; final evolved rubrics are used both to verify intermediate steps and to evaluate agent solo outputs.
• Empirical results (summary):
  • Personalization yields 4.5–12.9% (context) and 4.5–20.9% (weight) gains in solo task success over sessions; improvements often converge within ~20 task sessions.
  • On held-out tasks tied to the same users, context and weight adaptations generalize (3.2–6.6% and 4.8–5.2% gains, respectively).
  • Evolved rubrics detect 16.0–22.3% more failures than rubrics created by LMs or humans alone.
  • Agents learn both shared community norms and user-specific preferences; cross-user transfer produces smaller but nontrivial gains (0.3–8.8% on shared items; 6.2–19.6% on personal items).

Data & Methods

• Participants & scale:
  • 30 human experts total (two domains × three adaptation scenarios × five users per scenario).
  • 600 tasks total: each human performed 20 tasks. Task families: paper abstract writing (given title + intro) and data visualization (create HTML visual from data + instruction).
  • Additional held-out datasets: 30 held-out tasks per domain for generalization testing.
• Agent backbone and implementation:
  • Backbone LM: Qwen3.6-35B. Adaptations performed with editable contexts and LoRA adapters.
  • Interface built on an agent coworking framework extended with Plan, Deliverables, Message, and Rubrics modules.
  • Agent/human action spaces formalized (messages, file_create/edit/read, execute, plan for agent; plus verify, context_edit, trigger for human).
• Context adaptation:
  • After each session, an LM is prompted to induce/update memory M (declarative preferences/facts) and skills K (procedural workflows) based on the human’s actions and edits, and these are prepended as context for future tasks.
• Weight adaptation:
  • Use DPO to fine-tune LoRA adapters. Preference pairs constructed by pairing the initial one-shot agent output with the consolidated human-refined final solution (trajectories merged into τ_A*). Augmentation: additional negative samples come from agent-sampled rejects during training.
  • Reference policy set to the initial policy; β controls drift.
• Evaluation:
  • Solo-agent success: first final-step artifact produced solely by the agent (first iteration) evaluated against the evolved rubrics derived from the same user’s interaction stream.
  • Rubric quality: measured by how many failures the rubric catches on imperfect agent outputs, compared to LLM-alone and human-alone rubrics.
  • Generalization: performance on held-out tasks tied to the same users.
• Key design choices: enable users to edit rubrics/verifications so the verifier evolves iteratively; consolidate intermediate edits into single-shot targets for weight training so agents learn to produce preferred outputs without relying on further interaction.

Implications for AI Economics

• Productivity and value capture:
  • Personalization improves first-pass quality and reduces iterative editing costs for professionals, increasing effective labor productivity and potentially raising per-task billing rates or enabling higher throughput.
  • Firms selling agent customization (context + LoRA adapters) can capture premium value from professionals who need reputation-grade outputs.
• Cost vs. benefit trade-offs:
  • Context adaptation (verbalized memory/skills) is cheap to store/serve but bounded by what can be articulated; weight adaptation (LoRA + DPO) internalizes tacit patterns and yields larger persistent gains but requires compute and model-update costs. Providers must price and provision these trade-offs.
• Platform design & monetization:
  • Multi-session interaction logs are high-value signals; platforms can monetize personalization services or offer subscription tiers for evolving agents and rubrics. This also creates incentives to collect and retain user interaction data.
• Labor-market effects:
  • Augmenting experts with personalized agents could increase demand for higher-level curation and oversight (complementarity), while reducing time spent on routine edits. For novices, the availability of high-quality personalized agents could compress skill-differentiation, with ambiguous effects on wage dispersion.
• Externalities, competition, and lock-in:
  • Personalized adapters and editable memories create switching costs and potential vendor lock-in, favoring dominant platforms that aggregate large interaction datasets.
  • Evolving rubrics encode user preferences that could amplify subjective biases; economic actors (employers, clients) may need standards for auditing or certifying rubric-driven evaluations.
• Privacy, ownership, and incentives:
  • Interaction traces contain sensitive tacit knowledge and may be proprietary (firm playbooks). Economic models should address who owns adaptation artifacts (context, LoRA), how to price them, and how to compensate users/providers for shared tuning data.
• Policy and regulation:
  • As agents increasingly incorporate user-specific evaluation criteria, regulators may need guidance on transparency (which preferences were encoded?) and fairness when agent outputs affect employment, credit, or regulated decisions.

Overall, TAHI demonstrates that relatively small amounts of structured multi-session interaction data can yield economically meaningful personalization gains. Platforms, vendors, and users should weigh the persistent benefits of weight updates against compute and privacy costs, and consider governance and pricing architectures that reflect the high value of interaction-derived personalization.