4 cumulative citations
View corpus contextLLM data agents automate routine, structured analysis effectively but stumble on unstructured problems; Claude-Sonnet-4.5 tops overall, while multimodal perception improves vision-task performance.
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
DSAEval shows modern multimodal LLM data agents perform well on structured analytics and routine workflows but struggle on unstructured (text/image) data, with Claude-Sonnet-4.5 best overall and multimodal perception producing clear gains on vision tasks.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., image and text). DSAEval incorporates three distinctive features: (1) Multimodal Environment Perception, which enables agents to interpret observations from multiple modalities, including text and vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across reasoning, code, and results. We systematically evaluate 13 recent advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance, MiMo-V2-Pro and GPT-5.2 lead in duration and step efficiency, respectively, and MiMo-V2-Flash is the most cost-effective. We further demonstrate that multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04\% to 11.30\%. Overall, while current data science agents perform well on structured data and routine data analysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions.
Summary
Main Finding
DSAEval is a large, realistic benchmark for evaluating data-science agents that (1) covers 641 real-world problems from 285 heterogeneous datasets (tabular, time series, text, image), (2) runs agents in a multimodal Jupyter sandbox that preserves state across multi-query sessions, and (3) evaluates agents along three open-ended dimensions (reasoning, code, final report) using LLM judges. Evaluating 13 state-of-the-art LLM/VLM agents, the paper finds: Claude-Sonnet-4.5 has the highest overall score (8.164), multimodal environment perception yields consistent gains on vision-related tasks (2.04%–11.30%), agents are strong on structured/tabular workflows but struggle on unstructured domains (CV/NLP) and unsupervised tasks (e.g., clustering), and there are clear trade-offs between performance, time/step efficiency, token consumption, and cost.
Key Points
-
Benchmark scale & composition
- 641 problems, 285 datasets, assembled from 2,000+ sources (Kaggle, textbooks, notebooks).
- Data modalities: tabular 79.1%, time series 9.1%, text 5.8%, images 4.2% (so heavily tabular-weighted).
- Task coverage: Data Analysis (54.3%), Data Preparation, EDA, Feature Engineering, Modeling, CV, NLP, Clustering, etc.
-
Evaluation design
- Sandbox environment: Jupyter kernel + persistent state + GPU support (4× NVIDIA A100 80GB).
- Observations include: text logs, table previews/serialized dataframes, and image encodings (plots/figures).
- Sessions = dataset + sequence of interdependent queries (max 20 interaction turns per session, 1-hour timeout per iteration).
- Final outputs: executed notebook + textual report.
-
Scoring & judges
- Multi-dimensional score: Soverall = 0.3·Sreason + 0.3·Scode + 0.4·Sresult (α=β=0.3).
- Judge models: initially considered Claude-Haiku-4.5, GPT-5.1, Doubao-Seed-1.8; final reporting used a validated dual-judge protocol via a Human-LLM Alignment Study.
- Soft ground truth: questions/reasoning/answers synthesized with advanced LLMs and human-validated (acceptance rate ~73.5%).
-
Empirical results (high-level)
- Top overall: Claude-Sonnet-4.5 (8.164), Mimo-v2-Pro (7.912), GPT-5.2 (7.713), MiniMax-M2.7 (7.699).
- Efficiency trade-offs: MiMo-V2-Pro best token efficiency; GPT-5.2 best step efficiency; MiMo-V2-Flash most cost-effective.
- Multimodal perception improved performance on vision-related tasks by 2.04%–11.30%.
- Strong performance on structured/tabular tasks and standard engineering workflows; weaknesses remain for unstructured CV/NLP and unsupervised clustering.
Data & Methods
-
Data curation
- Sources: 2,000+ open datasets and 50 textbooks; highest-voted Kaggle notebooks preferred.
- Filtering: removed short/low-quality/duplicate/unavailable cases.
- Synthetic reference (soft ground truth): advanced LLMs (GPT-5, Grok-4) generated Question–Reasoning–Answer (QRA) pairs; human experts validated the curated subset.
-
Sandbox & agent interaction
- Agents produce code per step; environment executes code and returns multimodal observation ot = {otxt, otab, oimg}.
- Persistent kernel maintains variables and state across steps; deep-learning tasks supported with GPU acceleration.
-
Evaluation protocol
- Multi-query sessions (reflecting iterative real workflows).
- Multi-dimensional LLM-based judges score reasoning, code, and final report (open-ended acceptance of alternative valid solutions).
- Efficiency metrics collected: duration (seconds), number of agent steps, token consumption, and monetary cost (pricing from OpenRouter).
-
Models evaluated
- 13 LLM/VLMs (closed-source and open): GPT-5.2, GPT-5-nano, Gemini-3-Pro, Claude-Sonnet-4.5, MiMo-V2-Pro/Flash, GLM-4.6V, MiniMax-M2(.7), DeepSeek-V3.2, Qwen3-VL-30b, Mistral-14B, Grok-4.1, etc.
Implications for AI Economics
-
Cost vs. capability trade-offs
- Procurement should consider not only peak model accuracy but step efficiency, token usage, and per-task end-to-end time; cheaper models can be more cost-effective (MiMo-V2-Flash), while some models give better marginal gains per token or per agent step.
- Deep-learning tasks require GPU-backed sandboxes; enterprise adoption implies additional capital & operational costs (GPU fleets, orchestration), shifting per-task economics vs. purely text-only agents.
-
Labor & productivity impacts
- High performance on structured/tabular workflows indicates immediate potential to automate routine data-wrangling, EDA, and model engineering tasks—likely reducing time spent by data analysts on repetitive work.
- Persistent gaps on unstructured problems (CV/NLP) and unsupervised tasks suggest humans remain essential for high-complexity or exploratory analytics—implying complementarity rather than full substitution in many settings.
-
Investment priorities
- Investing in multimodal VLM capability yields measurable returns on vision-related workflows (2–11% gains), so organizations with CV-heavy pipelines will see higher ROI from multimodal models and GPU infrastructure.
- Benchmarks are heavily tabular; firms whose workloads are less tabular should interpret benchmark-derived procurement signals cautiously and may need domain-specific evaluation.
-
Market & service implications
- There is demand for benchmark-as-a-service and validated judge pipelines (human-LLM alignment) because open-ended evaluation is costly; vendors that provide reliable, domain-aware evaluation could capture value.
- Soft ground-truth and LLM-judge biases mean procurement decisions based solely on benchmark ranks could be misaligned—buyers should request domain-representative pilots.
-
Suggested economic metrics to monitor
- Cost per completed session (including GPU, model tokens, orchestration).
- Time-to-answer and human-in-the-loop overhead per task (to compute productivity lift).
- Error/rollback rates on unstructured tasks (to estimate residual human QA costs).
- Marginal value of multimodality: delta in business KPIs for CV/NLP tasks post-adoption.
Overall, DSAEval provides a useful, realistic lens for comparing data-agent technologies and quantifying operational trade-offs. For economists and procurement decision-makers, the benchmark informs which parts of the data-science stack are ready for automation, what infrastructure investments matter, and how to evaluate cost-effectiveness beyond raw model accuracy.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides a systematic, multi-agent benchmark across 641 real-world tasks and 285 datasets, giving robust descriptive evidence about current agent performance; however it does not identify causal effects on economic outcomes, evaluation metrics and task selection may introduce bias, and results may not generalize as models and agents rapidly evolve.
Methods Rigormedium — Evaluation covers many agents, multimodal inputs, iterative interactions, and multi-dimensional scoring (reasoning, code, results), which is methodically thorough for a benchmark; nonetheless potential weaknesses include selection bias in tasks/datasets, possible subjectivity in scoring or aggregation, limited transparency about agent configurations/seed variability, and sensitivity to runtime/cost settings.
SampleBenchmark of 641 real-world data-science problems drawn from 285 diverse datasets spanning structured (tabular) and unstructured modalities (text, images); 13 recent agentic LLM systems were evaluated across multimodal perception, multi-query interaction workflows, and multi-dimensional metrics (reasoning, code, results), with additional operational measures (duration, step efficiency, cost).
Themesproductivity human_ai_collab
GeneralizabilityDataset selection may not represent the full diversity of industry data science workloads or domain-specific tasks, Benchmark tasks and scoring rubrics could favor certain agent architectures or toolchains, Evaluation is sensitive to specific agent versions, APIs, and runtime/cost configurations that change rapidly, Likely English- and Western-centric tasks/data; limited evidence on non-English or cross-cultural datasets, Simulated agent-only evaluation omits human-in-the-loop workflows and organizational integration constraints
Claims (12)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| DSAEval is a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., image and text). Other | positive | number_of_problems_and_datasets |
Reading fidelity
high
Study strength
high
|
n=641
|
| DSAEval incorporates Multimodal Environment Perception, enabling agents to interpret observations from multiple modalities including text and vision. Other | positive | multimodal_perception_support |
Reading fidelity
high
Study strength
high
|
n=641
|
| DSAEval incorporates Multi-Query Interactions to mirror the iterative and cumulative nature of real-world data science projects. Other | positive | support_for_multi_query_interaction |
Reading fidelity
high
Study strength
high
|
n=641
|
| DSAEval provides Multi-Dimensional Evaluation, offering holistic assessment across reasoning, code, and results. Other | positive | evaluation_dimensions_covered |
Reading fidelity
high
Study strength
high
|
n=641
|
| We systematically evaluate 13 recent advanced agentic LLMs using DSAEval. Other | neutral | number_of_agents_evaluated |
Reading fidelity
high
Study strength
high
|
n=13
|
| Claude-Sonnet-4.5 achieves the strongest overall performance on DSAEval. Output Quality | positive | overall_performance_score (aggregate across metrics) |
Reading fidelity
high
Study strength
medium
|
n=641
|
| MiMo-V2-Pro leads in duration efficiency (best duration) on DSAEval. Task Completion Time | positive | duration (time-to-complete tasks) |
Reading fidelity
high
Study strength
medium
|
n=641
|
| GPT-5.2 leads in step efficiency (fewest steps) on DSAEval. Task Completion Time | positive | step_efficiency (number_of_steps_to_solution) |
Reading fidelity
high
Study strength
medium
|
n=641
|
| MiMo-V2-Flash is the most cost-effective agent on DSAEval. Organizational Efficiency | positive | cost_per_task |
Reading fidelity
high
Study strength
medium
|
n=641
|
| Multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04% to 11.30%. Output Quality | positive | performance_on_vision_tasks (accuracy/score) |
Reading fidelity
high
Study strength
high
|
2.04% to 11.30%
|
| Current data science agents perform well on structured data and routine data analysis workflows. Output Quality | positive | performance_on_structured_data_tasks |
Reading fidelity
medium
Study strength
medium
|
n=641
|
| Substantial challenges remain for current data science agents in unstructured domains (e.g., image and text tasks). Output Quality | negative | performance_on_unstructured_data_tasks |
Reading fidelity
high
Study strength
medium
|
not reported
|