Post-training on long-horizon office workflows raises a Qwen3.5 agent’s SWE-Bench Pro pass@1 by 5.8 points, with matched-trajectory evidence that the model forms better local goals, builds and preserves task-relevant state, maintains higher-level constraints, and verifies results more often — suggesting long-horizon behavioral training yields domain-general gains.

Main Finding

Post-training a large language agent on long-horizon office/workflow tasks (LHMTA) improved its software-engineering performance despite the training set containing no software-engineering examples. Specifically, Qwen3.5-122B-A10B adapted on 363 LHMTA tasks increased pass@1 on SWE-Bench Pro by 5.8 percentage points. Paired trajectory analyses indicate the gains stem from improved “goal-directed execution” (GDE): better goal formation, state construction, goal stability, and verification — behavioral capabilities that generalize across domains.

Key Points

• Goal-Directed Execution (GDE): operationalized as four observable capabilities
  • Goal formation: deriving correct immediate targets from parent goals and working state.
  • State construction: gathering/interpreting/integrating environment information relevant to goals.
  • Goal stability: preserving higher-level constraints while pursuing lower-level work.
  • Verification: checking and obtaining evidence that goals (intermediate and final) are satisfied.
• Transfer hypothesis: long-horizon tasks exercise recursive goal loops and associated behaviors (GDE). Strengthening GDE in one domain (office workflows) can transfer to other domains (software engineering).
• LHMTA task demands targeted: deep decomposition, parallel investigation & synthesis, entangled constraints, and long dependent chains — each stressing parts of GDE.
• Empirical results:
  • +5.8 percentage points pass@1 on SWE-Bench Pro after post-training on LHMTA.
  • Matched trajectory comparisons show measurable gains on all four GDE behaviors in both office and software tasks.
  • Aggregate behavioral changes: similar retrieval volume but less repetition, patches more overlapping with reference implementations while adding fewer lines, and nearly double the share of trajectories that ran formal tests.
• Interpretation caveat: the causal link between long-horizon structure and cross-domain transfer is supported but remains a hypothesis; results come from a single model/experiment.

Data & Methods

• Source training collection (LHMTA):
  • 363 long-horizon office and workflow tasks used for post-training (40 additional tasks reserved for in-distribution evaluation).
  • 27 task categories: documents, spreadsheets, slides, web research, file management, calendar/scheduling, browser automation, planning, and office services.
  • Tasks exposed as RL-style environments via Model Context Protocol tools; deterministic Python graders provided strict pass/fail with criterion-level partial credit.
  • Typical successful trajectories: 30–40 tool-calling turns and 80k–100k tokens.
  • Tasks intentionally contained no software-engineering examples.
• Base model and adaptation:
  • Qwen3.5-122B-A10B (mixture-of-experts; ~122B total with ≈10B active parameters).
  • Adaptation via LoRA targeted to attention and MLP projections.
  • Training included a supervised warm-up (3,000 trajectories generated by Kimi K2.6 on LHMTA) prior to adaptation (paper text truncates mid-description of later training specifics).
• Evaluation:
  • Out-of-domain target: SWE-Bench Pro (software engineering benchmark).
  • Primary metric cited: pass@1 improvement of 5.8 percentage points.
  • Analysis consisted of:
    • Matched trajectory comparisons (paired pre/post trajectories) scored for the four GDE behaviors.
    • Aggregate behavioral statistics on retrieval, edit characteristics (overlap with reference, lines added), and verification behaviors (test execution).
• Conceptual framing:
  • Execution modeled as recursive goal loops (goal → action → update working state → verify) aligned with classic TOTE and cognitive architectures, enabling behavioral measurement of internal capabilities without assuming explicit symbolic internal architectures.

Implications for AI Economics

• Data selection and ROI:
  • Value of post-training data can depend more on structural/task complexity than domain similarity. Investing in diverse long-horizon, tool-rich task collections may yield cross-domain performance gains and higher ROI than collecting large volumes of narrowly domain-matched examples.
  • Firms and model providers should consider allocating annotation/training budgets toward curated, structurally rich workflows (deep decomposition, multi-source synthesis, long dependency chains) to get broader payoff across product lines.
• Model procurement and product strategy:
  • Buyers seeking software-engineering improvement may not require large domain-specific corpora; general long-horizon workflow fine-tuning can deliver measurable gains. This can lower marginal cost of customization and speed deployment.
  • For foundation-model vendors, including long-horizon, multi-tool workflow data in post-training mixes is a strategic lever for improving agentic capabilities that generalize across enterprise tasks.
• Labor and productivity:
  • Because behavioral capabilities (goal formation/state bookkeeping/verification) underlie many knowledge-work tasks, improvements via long-horizon training could raise productivity across occupations (not just the training domain). This shifts where automation pressure and complementarity effects might appear: gains in software engineering could be achieved indirectly via investments in other workflow data.
• Benchmarking and procurement metrics:
  • Behavioral metrics like GDE (goal stability, state construction, verification) provide actionable evaluation axes beyond aggregate accuracy. Procurement and benchmarking practices should incorporate such long-horizon behavioral evaluations to estimate real-world productivity impacts.
• Risk and uncertainty / further research needs:
  • Results derive from a single model and training run; causal attribution to specific long-horizon structural features is suggestive but not definitive. Economic decisions relying on these findings should factor replication risk.
  • More systematic experiments are needed to quantify marginal returns to different kinds of post-training data (length vs. diversity vs. structural complexity) and to map costs to realized productivity gains in field deployments.

If you’d like, I can: - Translate these findings into a short memo for executives about where to invest in training data, - Produce a back-of-envelope ROI model comparing domain-specific labeling vs. investing in long-horizon workflow datasets, - Or extract example behavioral evaluation prompts to measure GDE in your own model fleet.