2 cumulative citations
View corpus contextMarket-style auctions let small language-model agents shoulder complex tasks: SALE cuts dependence on the largest model by over half and trims compute cost by a third while matching or exceeding the top model's pass@1 on search and coding benchmarks.
Scaling Small Agents Through Strategy Auctions
SALE, a marketplace-inspired auction framework that has agents bid short strategic plans, coordinates heterogeneous small language-model agents to cut reliance on the largest model by 52%, lower overall cost by 35%, and improve pass@1 on deep search and coding tasks with minimal overhead.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Small language models are increasingly viewed as a promising, cost-effective approach to agentic AI, with proponents claiming they are sufficiently capable for agentic workflows. However, while smaller agents can closely match larger ones on simple tasks, it remains unclear how their performance scales with task complexity, when large models become necessary, and how to better leverage small agents for long-horizon workloads. In this work, we empirically show that small agents' performance fails to scale with task complexity on deep search and coding tasks, and we introduce Strategy Auctions for Workload Efficiency (SALE), an agent framework inspired by freelancer marketplaces. In SALE, agents bid with short strategic plans, which are scored by a systematic cost-value mechanism and refined via a shared auction memory, enabling per-task routing and continual self-improvement without training a separate router or running all models to completion. Across deep search and coding tasks of varying complexity, SALE reduces reliance on the largest agent by 52%, lowers overall cost by 35%, and consistently improves upon the largest agent's pass@1 with only a negligible overhead beyond executing the final trace. In contrast, established routers that rely on task descriptions either underperform the largest agent or fail to reduce cost, often both, underscoring their poor fit for agentic workflows. These results suggest that while small agents may be insufficient for complex workloads, they can be effectively "scaled up" through coordinated task allocation and test-time self-improvement. More broadly, they motivate a systems-level view of agentic AI in which performance gains come less from ever-larger individual models and more from market-inspired coordination mechanisms that organize heterogeneous agents into efficient, adaptive ecosystems.
Summary
Main Finding
Small, cheap agent models (4B–14B) match large models on simple tasks but fail to scale with task complexity. A marketplace-style routing mechanism—Strategy Auctions for Workload Efficiency (sale)—lets heterogeneous agents bid short strategic plans; scoring bids by predicted cost and value and using an auction memory for bid refinement yields substantial efficiency and accuracy gains. sale both reduces reliance on the largest agent and lowers overall inference spend while slightly improving pass@1 relative to the largest single agent.
Key Points
- Task complexity measured by human solution time (τ): small agents perform near-parity with large agents on very short tasks (≈87% pass@1 for deep search, ≈92% for coding) but fall far behind on long-horizon tasks (≈25% and ≈17% respectively).
- Larger models are more capable on complex tasks, and do not reliably produce much shorter execution traces that would justify their higher per-token cost.
- sale: agents submit short strategy plans as bids. Bids are scored by:
- Cost Ct,i = wc · π(ai) · |st,i| (price per M tokens × strategy length)
- Value Vt,i = wh·H(st,i) + Σj wj·γj(st,i) (entropy + peer-assessed quality)
- Selection minimizes cost − value; provisional winners trigger cheaper agents to refine bids using an auction memory; final winner’s plan is executed.
- sale is lightweight (no separate trained router required), works with off-the-shelf agents, adds negligible execution-token overhead beyond the final trace, and implements test-time continual improvement via a shared auction memory.
- Empirical results (deep search + coding): sale improves or matches the largest agent’s pass@1 (+3.5% deep search, +2.7% coding), reduces reliance on the largest agent substantially (paper reports reductions on the order of ~52% aggregate; domain-specific offloading reported as −65% deep search and −40% coding), and reduces total spend (reported −42% on deep search, −25% on coding). Exact numbers vary by domain but consistently show better performance–cost Pareto frontiers.
- Conventional routers that choose models from task descriptions either fail to improve accuracy or to reduce cost in agentic, long-horizon settings.
Data & Methods
- Benchmarks/domains: deep search (SimpleQA, PopQA, HotpotQA, GAIA, Humanity’s Last Exam subset) and coding (MBPP, LeetCode + some custom low-complexity MCQs).
- HST-Bench: a new human-timed dataset of 753 tasks. Task complexity τ(t) = average expert solution time (three annotators; Krippendorff’s α = 0.86). Tasks binned into five time bands: ≤6s (0–0.1 min), ≤30s (0.1–0.5), ≤2.5min (0.5–2.5), ≤12.5min (2.5–12.5), ≤60min (12.5–60).
- Models: Qwen3 family (4B, 8B, 14B, 32B). Greedy decoding. Effective price-per-million-tokens π: $0.05 (4B), $0.09 (8B), $0.16 (14B), $0.36 (32B).
- Evaluation: pass@1, scored with LLM-as-judge against ground truth. Trace/token usage measured to compare per-task compute.
- sale specifics:
- Agents produce short strategies st,i (prompts in Appendix).
- Cost proxy uses strategy length and per-token price.
- Value uses plan entropy and peer scoring (integer 0–5 from other agents).
- Auction memory stores past bids + outcomes; cheaper agents may retrieve and refine prior successful strategies before final selection.
- Selection criterion: minimize Ct,i − Vt,i (tunable weights wc, wh, wj).
- Baselines: single-agent deployment (each model), non-predictive ensembling (impractical for long traces), predictive routers trained on task descriptions (expensive and brittle).
Implications for AI Economics
- Marginal returns to larger parametric models are task-dependent. For long-horizon, high-value tasks, larger models remain necessary; for many short tasks, smaller agents are cost-effective. Thus procurement and deployment should be dynamic and task-aware rather than “one-size-fits-all.”
- Coordination mechanisms (market-like auctions) can expand the effective capacity of small, cheap agents without retraining them: test-time allocation + shared experience (auction memory) produce durable cost savings and improved utilization.
- From a cost-optimization perspective, sale demonstrates that system-level orchestration can shift the Pareto frontier more effectively than simply investing in larger models or in expensive routing-model training.
- Mechanism-design considerations: auctions expose strategic behavior possibilities. Practical deployments will need robust scoring, incentives for truthful plan proposals, and safeguards against gaming (the paper uses peer scoring + entropy to mitigate some issues).
- Operational economics: sale’s lightweight, model-agnostic routing lowers the barrier to integrating heterogeneous models (including third-party or spot-priced models), which supports diversification of compute sourcing and granular pricing strategies.
- R&D investment trade-offs: resources spent on market/coordination layers (auction design, memory infrastructure, peer assessment tooling) may yield higher ROI than further upscaling single-model capacity in many real-world workloads.
- Broader labor-market analogies: treating agents as freelancers implies emergent labor-like dynamics (reputation, upskilling via repeated wins). This raises governance and regulatory questions as agent ecosystems acquire economic value and incentives.
- Limitations and open economic questions: how to design truthful, robust auctions at scale; externalities when many tasks share memory; long-term incentives for model providers; and generalization of sale to other domains (beyond deep search and coding) remain areas for future work.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic empirical experiments showing consistent cost and performance gains from the SALE auction framework across multiple deep-search and coding benchmarks, reporting quantitative metrics (e.g., 52% reduction in reliance on the largest agent, 35% cost reduction, improved pass@1). However, evidence is limited to selected task families, particular model sizes and configurations, and simulated auction settings rather than real-world production deployments; statistical robustness, sensitivity to hyperparameters, and broader external validity are not fully established.
Methods Rigormedium — The authors evaluate across tasks of varying complexity, compare to plausible baselines (including established routers), and measure both cost and accuracy metrics; they also introduce mechanisms like shared auction memory and cost-value scoring. But the methods omit some important rigor features or do not fully report them (e.g., exact model families and sizes, statistical tests, ablation breadth, hyperparameter sweeps, latency/throughput trade-offs, and real-world deployment constraints), limiting reproducibility and the strength of causal claims about mechanism components.
SampleEmpirical experiments on 'deep search' and coding task benchmarks of varying complexity, using a heterogeneous pool of language-model agents of different sizes; evaluations report pass@1 accuracy, per-task routing decisions, reliance on the largest agent, and aggregated compute/cost metrics; baselines include task-description routers and running the largest agent to completion.
Themesorg_design productivity
GeneralizabilityResults tested only on specific deep-search and coding benchmarks; other task families (e.g., open-ended reasoning, dialogue, multi-modal) may behave differently, Findings depend on the particular model families, sizes, cost assumptions, and compute/pricing environment used in experiments, Simulated auction and shared-memory setup may not capture engineering, latency, and reliability constraints of production systems, Short-horizon experimental traces may not reflect long-run dynamics, adversarial tasks, or human-in-the-loop workflows, Performance gains may not scale identically with substantially larger model pools or different agent architectures
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Small agents' performance fails to scale with task complexity on deep search and coding tasks. Output Quality | negative | agent performance as task complexity increases |
Reading fidelity
high
Study strength
medium
|
not reported
|
| SALE (Strategy Auctions for Workload Efficiency) reduces reliance on the largest agent by 52% across deep search and coding tasks of varying complexity. Task Allocation | positive | reliance on the largest agent |
Reading fidelity
high
Study strength
medium
|
52%
|
| SALE lowers overall cost by 35%. Organizational Efficiency | positive | overall cost |
Reading fidelity
high
Study strength
medium
|
35%
|
| SALE consistently improves upon the largest agent's pass@1 with only a negligible overhead beyond executing the final trace. Output Quality | positive | pass@1 (correctness of top-1 output) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Established routers that rely on task descriptions either underperform the largest agent or fail to reduce cost, often both. Organizational Efficiency | negative | performance relative to largest agent and achieved cost reduction |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Smaller agents can closely match larger ones on simple tasks. Output Quality | positive | agent performance on simple tasks |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Small agents alone may be insufficient for complex workloads, but they can be effectively 'scaled up' through coordinated task allocation and test-time self-improvement. Organizational Efficiency | mixed | ability to handle complex workloads via coordination and self-improvement |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| In SALE, agents bid with short strategic plans scored by a systematic cost-value mechanism and refined via a shared auction memory, enabling per-task routing and continual self-improvement without training a separate router or running all models to completion. Task Allocation | positive | per-task routing efficiency and model usage (no separate router/tracing all models) |
Reading fidelity
high
Study strength
medium
|
not reported
|