0 cumulative citations
View corpus contextModels that win at doing the work often lose at coaching it: a new benchmark finds automation rankings poorly predict which LLMs improve a weaker worker’s output, and guidance from stronger models can be unhelpful or even detrimental across several real-world tasks.
CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks
Across seven professional tasks, LLM rankings differ substantially between automation and augmentation: the best autonomous models are often not the best assistants, and assistance can sometimes harm rather than help a weaker worker.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.
Summary
Main Finding
Automation ability is an incomplete proxy for assistance quality. When the same frontier LLMs are evaluated both as autonomous task solvers and as assistants to a fixed, lower-capacity worker, model rankings diverge substantially across roles and tasks: the best automator is often not the best assistant, assistance is frequently non‑beneficial, and which model is best depends on the task and role.
Key Points
- Benchmark design: CentaurBench evaluates models in two modes on seven real-world tasks — automation (model produces deliverable) and augmentation (model writes process-oriented assistance; a fixed worker executes).
- Tasks: counseling, market trends analysis, meal planning, operations research, tax preparation, travel planning, tutoring.
- Models evaluated (10): Claude-Opus-4.8, Claude-Sonnet-4.6, Gemini-3.1-Pro, DeepSeek-V3.1, GPT-5-Mini, GPT-OSS-120B, GPT-O4-Mini, GPT-4.1, GPT-O3-Mini, GPT-3.5-Turbo.
- Worker model: GPT-3.5-Turbo is used as the standardized lower-capacity executor to isolate assistant guidance value.
- Evaluation: blind pairwise comparisons by a panel of four LLM judges using task-specific micro‑rubrics (1–10 scales), with leave‑one‑family‑out masking to avoid self‑preference; pipeline replicated across ten independent runs.
- Quantitative highlights:
- Model-level rank correlation between augmentation and automation ≈ 0.48 (moderate), but varies by task from −0.04 (travel planning) to 0.85 (tax preparation).
- Automation winner loses augmentation on 5 of 7 tasks.
- Assistance often not positive: the unaided worker (GPT-3.5-Turbo) outranks every assisted condition on three tasks.
- Only one model’s guidance (GPT-5‑Mini) beats no-guidance on average; GPT-5‑Mini ranks first in both modes overall.
- Example reversals: Claude‑Opus‑4.8 is a top automator on market trends (mean rank 2.05) but among weakest assistants there (8.15); GPT‑4.1 is a relatively weaker automator but a strong assistant on counseling.
- Reproducibility: code and interactive dashboard available (authors’ GitHub and dashboard links).
Data & Methods
- Framework: Two usage modes per model/task pair:
- Automation: focal model produces final deliverable.
- Augmentation: focal model produces an assistance text (requirements check, execution plan, final checklist); GPT‑3.5‑Turbo (fixed) produces deliverable using that guidance. Assistants are explicitly forbidden from writing the final deliverable.
- Task selection: Seven heterogeneous, economically grounded tasks chosen to vary in structure, domain knowledge, risk, and human-facing judgment. Each task has an observable deliverable and a micro‑rubric aligned to prompt requirements.
- Rubrics & scoring: Each task has task-specific dimensions (e.g., math correctness, dietary safety) plus general dimensions (instruction-following, usefulness, organization, tone). Judges perform blind pairwise comparisons guided by rubrics; rubric scores support rationale though winners are chosen contrastively.
- Judge masking and replication: Judges are LLMs; judges are barred from evaluating outputs from their own model family (leave‑one‑family‑out). Entire evaluation pipeline replicated ten times to assess stability.
- Aggregation: Pairwise outcomes aggregated to produce rank matrices by task and usage mode; comparisons report mean ranks across runs and statistical stability of reversals.
Implications for AI Economics
- Model procurement and deployment should be role-specific. Organizations choosing LLMs must match model choice to the intended role (assistant vs. automator) and the specific task — a single leaderboard of autonomous performance is insufficient for deployment decisions.
- Labor and organizational effects depend on role allocation. Automation‑oriented procurement will have different impacts on labor demand and task allocation than augmentation‑oriented deployments. Misaligned model-role choices can reduce worker productivity or worsen outcomes.
- Policy and regulation: Evaluations that inform policy (e.g., safety, certification, procurement standards) should require role-aware testing. Claims about “best model” should be qualified by the role and task context.
- Evaluation practice: Benchmarks and procurement tests should measure augmentation quality explicitly (marginal guidance value), not only autonomous task performance. Leave‑one‑family-out masking and role‑specific rubrics are useful methodological elements.
- Research directions: study heterogeneity across worker skill levels (human workers vs. different-model workers), dynamic/adaptive assistance, team composition (multi-agent roles), and economic tradeoffs (cost vs. role performance). Field experiments and labor-market analyses should distinguish between augmentation and automation effects on productivity, wages, and task allocation.
- Risk management: Assistance can harm downstream performance if guidance is miscalibrated or overly complex; firms should validate assistive deployments before scaling and consider simpler, worker‑matched guidance as part of design.
Summary conclusion: CentaurBench demonstrates that “best automator” ≠ “best coach.” Economics of AI adoption must account for role-specific model capabilities, and empirical evaluation frameworks should explicitly measure augmentation alongside automation.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — Strong internal design for isolating the assistant’s marginal contribution (fixed worker, rubrics, blind pairwise LLM judging, masking, replication). However, external validity is limited because the worker and judges are LLMs (not human workers), tasks are a small curated set (7), and the model set is constrained to ten models; evaluation therefore demonstrates important benchmarking insights but does not directly measure human productivity or field outcomes.
Methods Rigorhigh — Careful experimental benchmarking: clear task selection with micro-rubrics, paired automation/augmentation regimes, a fixed low-capacity worker to identify marginal guidance value, blind pairwise comparisons, leave-one-family-out judge masking, and ten independent replications; main limitations are reliance on LLM judges and an artificial model–model interaction rather than human-in-the-loop trials.
SampleBenchmark of 10 frontier LLMs (Claude-Opus-4.8, Claude-Sonnet-4.6, Gemini-3.1-Pro, DeepSeek-V3.1, GPT-5-Mini, GPT-OSS-120B, GPT-O4-Mini, GPT-4.1, GPT-O3-Mini, GPT-3.5-Turbo) evaluated across 7 real-world tasks (counseling, market analysis, meal planning, operations research, tax preparation, travel planning, tutoring). Augmentation trials use GPT-3.5-Turbo as the fixed worker; each model also runs in automation mode producing deliverables directly. Outputs are scored via blind pairwise comparisons by a panel of four LLM judges (with family masking), using task-specific micro-rubrics; the full pipeline is replicated across 10 independent runs.
Themeshuman_ai_collab productivity
IdentificationMeasures the marginal value of model-generated guidance by holding the downstream worker constant (GPT-3.5-Turbo) and comparing outcomes when a focal model either (a) produces the deliverable directly (automation) or (b) supplies an assistance text that the fixed worker uses to produce the deliverable (augmentation). Outputs are evaluated via blind, rubric-guided pairwise comparisons by a panel of LLM judges (with leave-one-family-out masking) and replicated across ten runs to isolate assistant quality from executor ability.
GeneralizabilityResults are from model–model interactions (assistant model → GPT-3.5 worker) and may not generalize to human workers or to higher-capability human-model teams., LLM judges (rather than human raters) may introduce systematic evaluation biases correlated with model families or rubrics., Only seven curated tasks; other tasks, industries, or workflows could show different augmentation/automation relationships., Model set is limited in coverage and time-specific (ten models available at time of study); future or other models may behave differently., Prompt and rubric design choices (including the assistance-text constraint) shape outcomes and may advantage or disadvantage particular model reasoning styles.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Model rankings in automation and augmentation are only modestly correlated, with a model-level rank correlation of about 0.48. Output Quality | mixed | Rank correlation between model performance as autonomous task solvers and as assistants |
Reading fidelity
high
Study strength
medium
|
n=10
about 0.48 rank correlation
|
| The correlation between automation and augmentation rankings varies substantially across tasks, ranging from -0.04 for travel planning to 0.85 for tax preparation. Output Quality | mixed | Task-specific rank correlation between automation and augmentation performance |
Reading fidelity
high
Study strength
medium
|
n=7
−0.04 to 0.85
|
| The model that wins in automation loses in augmentation on five of the seven tasks. Output Quality | mixed | Whether the top automation model is also the top augmentation model on each task |
Reading fidelity
high
Study strength
medium
|
n=7
5 of 7 tasks
|
| Assistance from another model is not reliably beneficial: the unaided GPT-3.5-Turbo worker ranks above every assisted condition on three tasks. Output Quality | negative | Final deliverable quality under assisted versus unaided worker conditions |
Reading fidelity
high
Study strength
medium
|
n=7
3 tasks
|
| Only one assistant model's guidance outperforms providing no guidance on average. Output Quality | negative | Average final-output ranking with model guidance versus no guidance |
Reading fidelity
high
Study strength
medium
|
n=10
only one model
|
| GPT-5-mini ranks first in both the automation and augmentation comparisons, while the unaided GPT-3.5-Turbo worker has the second-best average rank in the augmentation comparison. Output Quality | mixed | Average rank of models and baseline conditions in automation and augmentation |
Reading fidelity
high
Study strength
medium
|
n=10
first in both regimes; second-best average rank for the unaided worker
|
| On the market trends analysis task, Claude-Opus-4.8 ranks highly as an autonomous solver but among the weakest assistants. Output Quality | negative | Mean rank of Claude-Opus-4.8's final outputs when solving directly versus providing assistance |
Reading fidelity
high
Study strength
medium
|
n=10
mean rank 2.05 in automation versus 8.15 in augmentation
|
| GPT-4.1 displays the reverse pattern on counseling: it is a lower-ranked direct solver but the strongest assistant. Output Quality | mixed | Relative rank of GPT-4.1 as a direct counseling-response generator versus an assistant |
Reading fidelity
high
Study strength
medium
|
n=10
|
| The value of assistance depends on both the assistant model and the task, so a single general-purpose model leaderboard may be inadequate for organizational model selection. Task Allocation | mixed | Variation in assistance quality across models and professional tasks |
Reading fidelity
high
Study strength
medium
|
n=7
|