Repurposing large language models for sequential decision-making yields large benchmark gains and better simulated bidding: a 3B-parameter DecisionLLM substantially outperforms Decision Transformer on Maze2D and improves AuctionNet bidding performance, with results driven by model scale, dataset size and trajectory quality, though findings are limited to offline and simulated settings.

Main Finding

DecisionLLM is a multimodal Transformer that treats trajectories (states, actions, returns-to-go) as a distinct non-text modality and aligns them with textual task descriptions inside an LLM. This design substantially improves long-horizon offline decision-making: embedding analyses show better, disentangled representations than prompt-based text serializations, and empirical results report large gains over Decision Transformer (e.g., ≈+69.4 points on Maze2D umaze-v1 and an improvement on AuctionNet reported in the paper as roughly +0.06–0.085). The authors also identify scaling laws: performance improves with model size, dataset volume, and data quality, and these factors interact synergistically.

Key Points

• Problem: LLMs treat numeric trajectory values as text tokens and are insensitive to numeric magnitude/order; naive prompting of continuous trajectories is ineffective for decision tasks.
• Core idea: treat trajectories as a separate modality via a trajectory encoder; fuse its embedding with text embeddings via placeholder tokens (<|traj begin|>, <|traj end|>) and autoregressively predict continuous actions through an action head.
• Architecture:
  • Trajectory encoder converts returns-to-go, states, and actions into modality-specific embeddings, interleaved by timestep.
  • Trajectory embedding replaces placeholder token embeddings in the LLM input.
  • LLM is finetuned end-to-end; a linear action head maps outputs back to continuous action space; training uses MSE.
• Training & inference:
  • Autoregressive training on offline trajectories; sliding-window for long trajectories.
  • Inference iteratively conditions on predicted action, observed next state and updated returns-to-go.
• Data quality handling:
  • Filter out low-return trajectories.
  • Reweight low-reward steps instead of outright removal to preserve coverage.
• Empirical signals:
  • Prompt-based numeric encoding shows representation collapse (high pairwise similarity); DecisionLLM embeddings are structured and clusterable.
  • DecisionLLM-3B substantially outperforms Decision Transformer on maze2d-umaze-v1; achieves measurable gains on AuctionNet (paper reports inconsistent numeric values; improvements are small-to-moderate on auction benchmark).
• Scaling laws: model size, data volume, and data quality all matter and interact — scaling one dimension alone has limited effect without the others.
• Practical costs: approach requires finetuning large pretrained LLMs (authors finetuned Qwen2.5-Instruct variants) and uses substantial compute (A100s).

Data & Methods

• Tasks / Benchmarks:
  • Maze2D (D4RL; long-horizon, sparse reward navigation; primary environment maze2d-umaze-v1).
  • AuctionNet (simulated bidding benchmark to test applicability in computational advertising).
• Baselines compared: BC, TD3+BC, CQL, IQL, Decision Transformer (DT), prompt-based LLM input.
• Key modelling details:
  • Initialize from Qwen2.5-Instruct pretrained weights at multiple scales; finetune entire model including input/output linear layers and LLM parameters.
  • Trajectory input: interleaved triplets (returns, state, action) embedded and positionally encoded.
  • Fusion: text tokens include placeholders; trajectory embedding substitutes placeholder embeddings.
  • Loss: MSE between predicted and ground-truth continuous actions.
  • Sliding window over last t timesteps (t=20 window used in experiments).
• Training hyperparameters (as reported):
  • Batch size 64, learning rate 1e-5, training 5 epochs, evaluation every 200 steps, cosine annealing LR schedule.
  • Hardware: 8× NVIDIA A100 (40GB).
• Data quality interventions:
  • Return-threshold filtering to remove low-quality trajectories.
  • Reweighting of low-reward steps to preserve exploration coverage.
• Evaluation:
  • Maze2D metrics: return and normalized D4RL score; results averaged over 100 runs; they report peak checkpoint performance.
  • AuctionNet: online scoring with 48 players and 7 episodes per evaluation.

Implications for AI Economics

• Practical application to markets (computational advertising / RTB):
  • DecisionLLM extends AI-Generated Bidding (AIGB) by enabling LLM-based agents to make long-horizon strategic bidding decisions that condition on trajectories and textual campaign objectives.
  • Better modeling of continuous bid and state dynamics can increase bid efficiency and seller revenue capture or buyer ROI depending on objective alignment.
• Market structure and competitive effects:
  • The identified scaling law (model size × data volume × data quality) implies incumbents with larger high-quality datasets and compute budgets can gain outsized advantages in bidding efficiency, potentially increasing market concentration.
  • Firms that can collect/clean high-quality offline trajectory data and fine-tune large LLMs will be better positioned to deploy superior automated bidding strategies.
• Welfare, efficiency and strategic behavior:
  • Improved long-horizon agents may change auction dynamics (e.g., temporal bidding strategies, pacing), with ambiguous welfare effects: could improve allocative efficiency but also enable more aggressive extraction or exploit complex auction features.
  • Multi-agent interaction: when many bidders adopt such agents, new equilibria may emerge; simple one-agent offline evaluation may not predict market outcomes.
• Risks and limitations relevant to economics:
  • Offline imitation-learning sensitivity: DecisionLLM is trained offline and can inherit biases/limitations of historical data (e.g., selection bias, lack of coverage for strategic deviations), leading to suboptimal or unstable behavior in live markets.
  • Distributional shift: the model may overfit to historical auction environments; robust online adaptation or safe exploration is needed to avoid performance collapses.
  • Cost externalities: high compute and data costs raise barriers to entry and could exacerbate inequality across firms.
  • Regulatory/privacy concerns: richer trajectory data and user-level signals used to tune models raise privacy and regulatory considerations.
• Research & policy directions:
  • For researchers: evaluate DecisionLLM in multi-agent simulated markets to study equilibrium dynamics, adversarial behaviors, and emergent pacing strategies; explore robust online fine-tuning and off-policy correction methods.
  • For economists and platform designers: assess how improved bidder agents alter auction outcomes, platform revenue, advertiser surplus, and consumer welfare; consider mechanisms (reserve pricing, throttling, transparency rules) to mitigate concentration or exploitative tactics.
  • For practitioners: invest in high-quality trajectory collection and cleaning; consider cost-benefit trade-offs of scaling model size vs. improving data quality; deploy conservative online testing (A/B tests, sandboxed simulations) before production rollout.

If you want, I can extract the exact experimental tables and numeric results from the paper, highlight the inconsistencies in reported AuctionNet gains, or draft a short briefing for product or policy teams focusing on market impact and mitigation strategies.