Shaping the execution environment, not just agent code, substantially increases LLM agents' ability to autonomously discover scientific solutions: EurekAgent's permission, artifact, budget, and human-in-loop engineering yields state-of-the-art results on multiple benchmarks and even finds a 26-circle packing solution for under $11 in API cost. The findings suggest that designing environments is a key lever for productive, low-cost autonomous research agents, though gains are shown on a narrow suite of tasks and depend on specific model/APIs.
EurekAgent: Agent Environment Engineering is All You Need For Autonomous Scientific Discovery
EurekAgent—an environment-engineered system that controls permissions, artifact management, budgets, and human oversight—enables LLM-based agents to autonomously propose, validate, and iterate discoveries, outperforming prior approaches on several math, kernel, and ML benchmarks while reporting low API costs.
LLM-based agents have shown increasing potential in automating scientific discovery. Given an optimizable metric and an execution environment, they can propose, validate, and iterate scientific solutions, and have produced results that outperform human-designed approaches. As model capabilities continue to improve, we argue that the bottleneck for autonomous scientific discovery is shifting from prescribing agent workflows to designing agent environments: the resources, constraints, and interfaces that shape agent behavior. We frame this as environment engineering: building environments that amplify productive behaviors, such as open-ended exploration, systematic artifact management, and inter-agent collaboration, while suppressing harmful behaviors, such as reward hacking and high-friction human oversight. We present EurekAgent, an environment-engineered agent system for metric-driven autonomous scientific discovery. EurekAgent engineers the environment along four dimensions: permissions engineering for bounded agent execution and isolated evaluation; artifact engineering for filesystem and Git-based collaboration; budget engineering for budget-aware exploration; and human-in-the-loop engineering for easy human supervision and intervention. EurekAgent sets new state-of-the-art results on multiple mathematics, kernel engineering, and machine learning tasks, including new state-of-the-art 26-circle packing results discovered with less than $11 in total API cost. We open-source our code and results, and call for environment engineering as a core research direction for developing reliable autonomous research agents.
Summary
Main Finding
EUREKAGENT shows that carefully engineering the agent environment — permissions, artifacts, budgets, and human oversight — is sufficient to enable off-the-shelf LLM-based CLI agents to perform state-of-the-art autonomous metric-driven scientific discovery. Without prescribing specialized agent workflows, EUREKAGENT achieved new SOTA results across several mathematics and kernel-engineering tasks (and top performance on an evaluated MLE-Bench subset) while keeping API costs very low (average < $17; 26-circle packing discovered with <$11).
Key Points
- Core argument: as base LLM agents become strong, the main bottleneck shifts from prescribing agent workflows to designing the environment that shapes agent behavior (environment engineering).
- EUREKAGENT architecture: a simple outer-loop of PREPARE → PROPOSE → IMPLEMENT (R rounds, up to P parallel implementations per round). The environment handles coordination, persistence, evaluation access, and budget enforcement; agent sessions decide strategies internally.
- Four environment-engineering dimensions:
- Permissions engineering: Docker sandbox, run-level isolation, hidden evaluator exposed only via secure grading API, same-round isolation between parallel implement sessions, controlled GPU acquisition API.
- Artifact engineering: shared filesystem + Git commits as long-term memory; ranked solution history; cached web-search logs; persistent run directories for traceability/resumability.
- Budget engineering: explicit wall-clock and API cost limits per stage; time-checker helper API; passive deadline injection; run abortion/preservation when cost limit hit; tracking of token usage (not exposed to agents).
- Human-in-the-loop engineering: terminal UI (session-level chat, inspect logs) and web monitor (score evolution, ranked solutions) for oversight and intervention.
- Safety/robustness measures: prevents evaluator leakage and score tampering, blocks same-round copying, controller-owned authoritative result files, GPU lock API to avoid contention.
- Empirical results (selected highlights from Table 1):
- 26-circle packing: EUREKAGENT 2.635999 (prev best AI 2.635986, prev best human ~2.634) discovered with < $11 API cost.
- Erdős’ minimum overlap: 0.380870 (improves prior AI/human).
- First autocorrelation inequality: 1.502861 (improves prior AI/human).
- Kernel engineering / TriMul runtime: 2005.03 µs (prev best AI 2247.78 µs).
- MLE-Bench subset: 85.71% (previous 71.43%).
- Implementation details: experiments used Claude Code as the CLI agent and GLM-5.1 as the base LLM; code and results are open-sourced.
Data & Methods
- Problem inputs: human-provided problem description, hidden evaluator (kept outside agent-visible workspace), submission spec, optional initial code, and user-configured time/API budgets plus R (rounds) and P (parallelism).
- System loop:
- Prepare stage (one-off): agent reads inputs, tests hidden evaluator via secure interface, installs/validates dependencies, writes a preparation summary artifact.
- Propose stage (each round): proposal agent inspects prior evidence and optionally web/browse tools to output up to P hypothesis manifests with implementation-ready descriptions.
- Implement stage (each round): P parallel implementation agent sessions, each in a separate workspace, iteratively implement and submit candidates via secure hidden-evaluator API; same-round peers are isolated; evaluated submissions are recorded and ranked automatically.
- Environment technical components:
- Docker containers per run, mounted run workspace, controller-owned authoritative result files (agents cannot modify).
- Hidden evaluation service accessible only through a submission API returning official scores; all evaluated submissions logged.
- Filesystem + Git for artifact persistence and traceable history; agents instructed to create informative commit messages.
- Time-checker and budget tracking APIs; token usage tracked centrally but not revealed to agents.
- Web monitor and terminal UI for human oversight.
- Evaluation domains: mathematics, kernel engineering, and machine-learning-engineering tasks (ResearchClawBench-style metric-driven problems and MLE-Bench subset).
- Baselines: prior published human and AI SOTA; experiments use off-the-shelf CLI agents rather than bespoke agent architectures.
Implications for AI Economics
- High ROI from environment engineering over model redesign:
- EUREKAGENT demonstrates large performance and cost-efficiency gains by investing in system/infrastructure design rather than bespoke agent algorithmic changes. For firms and research funders this suggests substantial returns from building tooling, evaluation infrastructure, and secure sandboxes.
- Cost signals and incentive design:
- Budget engineering (explicit API and time budgets, time-checker helpers) shows how embedding budget constraints in environments can shape agent behavior toward cost-aware exploration. This has implications for pricing strategies (API/token pricing, per-run billing) and internal cost-accounting for automated research.
- Value capture and markets for research tooling:
- As environments become the bottleneck, firms offering robust, auditable environment platforms (secure evaluators, artifact stores, GPU management APIs, human-in-the-loop dashboards) can capture value beyond base LLM providers. There is a potential market for “research-as-a-service” platforms that package these affordances.
- Compute demand vs. effici ency:
- Environment-level controls (GPU lock API, runtime budgets) can reduce wasteful compute use and allow tighter capacity planning. Conversely, increased autonomous exploration may raise aggregate compute demand if widely adopted—so pricing and allocation mechanisms for shared infrastructure matter.
- Risk externalities and regulatory considerations:
- EUREKAGENT mitigates reward hacking and evaluator leakage, but the paper highlights that agents can exploit weak evaluation protocols. This raises the need for auditable evaluations, standardized benchmarks, and possibly certification for automated research systems to prevent unreliable or fraudulent outcomes.
- Labor and research markets:
- The model where humans set directions/metrics and supervise while agents perform large-scale exploration may shift labor towards higher-level design, metric formulation, and oversight. This could increase productivity per researcher but also change skill demand (infrastructure & governance skills).
- Implications for funding and procurement:
- Grants and corporate R&D budgets might reallocate from incremental model improvements to environment/infrastructure investment because cheaper compute + good environment design produced strong empirical gains (e.g., SOTA discovered for <$11).
- Governance and auditing economics:
- Systems that persist artifacts, Git histories, and secure evaluation logs reduce verification costs for reviewers, journals, and funders. That lowers the transaction costs of auditing AI-driven discoveries and may accelerate trust/adoption.
- Strategic pricing & business models for LLM providers:
- If environment engineering is the differentiator, base-model providers might compete on transparent token pricing, usage metering, and partnerships with environment/platform providers rather than purely on marginal model accuracy.
Caveats and limits (economic perspective) - External validity: results depend on strong base models (Claude Code, GLM-5.1) and careful environment implementation; less-capable models may not reproduce gains. - Some environment protections can be costly to operate (secure evaluators, GPU management) and their maintenance/scale economics were not deeply measured. - Societal/regulatory externalities (misuse, over-automation) remain and could impose compliance costs.
Overall, EUREKAGENT reframes where to invest to accelerate autonomous research: relatively low-cost, high-leverage investments in environment design can yield outsized improvements in effectiveness, cost-efficiency, and auditability — with broad implications for how R&D budgets, market offerings, and regulation evolve.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides multiple empirical results and concrete cost measurements showing improved performance on several benchmark tasks and reports ablations of environment components, which supports the claim that environment engineering matters; however, the evidence is limited to a selected set of tasks and model/API versions, lacks formal causal identification or statistical uncertainty measures, and may be sensitive to task selection or engineering choices.
Methods Rigormedium — Authors appear to use systematic experiments, ablation studies, and open-sourced code/results, which strengthens credibility, but the paper does not report pre-registered protocols, randomized comparisons, confidence intervals or robustness across many model families/versions, and may omit details (seeds, hyperparameters) needed for full reproducibility.
SampleA curated suite of scientific-discovery-style benchmark tasks including mathematics problems, kernel engineering tasks, and machine-learning benchmarks (explicitly reporting a 26-circle packing result); experiments run LLM-based agents via API within the EurekAgent environment, measuring task performance and API cost; code and results are open-sourced by the authors.
Themesproductivity innovation human_ai_collab adoption
IdentificationComparative benchmarking within engineered environments: the authors run LLM-based agents in their EurekAgent environment across a set of tasks and compare performance (and API cost) to prior/alternative approaches and ablation variants; no randomized assignment, instrumental variables, or other formal causal identification strategies are used.
GeneralizabilityLimited set of benchmark tasks (math, kernel engineering, ML) — may not generalize to other scientific domains (biology, chemistry, experimental lab work)., Results depend on the specific LLMs and API pricing/versions used; improvements may change with different models or model updates., Performance may be sensitive to environment hyperparameters and engineering choices that could be tuned to these tasks (risk of overfitting)., Cost and compute scalability to larger, real-world scientific problems is untested., Reproducibility depends on availability of precise implementation details, seeds, and API conditions.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| LLM-based agents have shown increasing potential in automating scientific discovery: given an optimizable metric and an execution environment, they can propose, validate, and iterate scientific solutions, and have produced results that outperform human-designed approaches. Research Productivity | positive | autonomous scientific discovery performance |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The bottleneck for autonomous scientific discovery is shifting from prescribing agent workflows to designing agent environments (the resources, constraints, and interfaces that shape agent behavior). Research Productivity | positive | research direction / enabling factors for agent performance |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| Environment engineering can amplify productive behaviors (e.g., open-ended exploration, systematic artifact management, inter-agent collaboration) while suppressing harmful behaviors (e.g., reward hacking and high-friction human oversight). Research Productivity | mixed | agent behavior quality (productive vs. harmful behaviors) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| We present EurekAgent, an environment-engineered agent system for metric-driven autonomous scientific discovery. Other | positive | availability of an environment-engineered agent system |
Reading fidelity
high
Study strength
low
|
not reported
|
| EurekAgent engineers the environment along four dimensions: permissions engineering for bounded agent execution and isolated evaluation; artifact engineering for filesystem and Git-based collaboration; budget engineering for budget-aware exploration; and human-in-the-loop engineering for easy human supervision and intervention. Other | positive | system design features implemented |
Reading fidelity
high
Study strength
low
|
not reported
|
| EurekAgent sets new state-of-the-art results on multiple mathematics, kernel engineering, and machine learning tasks. Research Productivity | positive | task performance on mathematics, kernel engineering, and ML tasks |
Reading fidelity
high
Study strength
medium
|
not reported
|
| EurekAgent discovered new state-of-the-art 26-circle packing results with less than $11 in total API cost. Research Productivity | positive | 26-circle packing solution quality and API cost |
Reading fidelity
high
Study strength
medium
|
26-circle packing results discovered with less than $11 in total API cost
|
| We open-source our code and results. Other | positive | code and results availability / reproducibility |
Reading fidelity
high
Study strength
low
|
not reported
|
| Environment engineering should be treated as a core research direction for developing reliable autonomous research agents. Research Productivity | positive | research prioritization / agenda setting |
Reading fidelity
high
Study strength
speculative
|
not reported
|