A new benchmark builds fully synthetic, repeatable enterprise estates so LLM agents can be graded exactly; realism checks lifted fidelity from about 62% to 97% across 23 firms while nine models scored between 42% and 77% on a 33-question evaluation.
The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents
The paper introduces Era by Eon, a deterministic generator and simulator suite that creates complete fictional enterprise estates with computed ground-truth answers so LLM agents can be exactly and repeatably evaluated, reporting improved data realism (mean score 61.8 → 97.0) and model accuracies from 42.4% to 76.8% on a 33-question simulator test.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We present the Era by Eon Benchmark for evaluating LLM agents that use enterprise tools. The benchmark is built around a complete fictional company. It includes product simulators, company-specific internal databases, benchmark questions, and computed answer keys. Industry, company size, business model, application portfolio, and a seed define each company. One seeded entity graph supplies shared company data to simulators of Salesforce, Zendesk, Slack, Gong, and other products. A questionconditioned generator creates the schemas and records for internal databases. It takes shared entities, keys, and values from the same graph before generating database-specific facts. Both mechanisms therefore describe one consistent enterprise estate. Every expected answer is computed from the final records, so grading is exact. Design and answer-key checks validate the internal databases. A realism scorecard and adversarial detector validate the entity graph. Across 23 generated companies, the mean realism score rose from 61.8 to 97.0, with zero records flagged as synthetic. In the reported simulator-track comparison, nine models answered the same 33 questions three times each. Accuracy estimates ranged from 42.4% to 76.8%, and three of 36 pairwise differences remained supported after correction.
Summary
Main Finding
The Era by Eon Benchmark is a deterministic, fully generated evaluation environment for LLM agents that operate over enterprise systems of record. It produces a complete, internally consistent fictional enterprise (an entity graph), projects that graph into many product simulators (Salesforce, Zendesk, Slack, Gong, S3, Google Drive, etc.), generates company-specific internal databases from author-supplied business questions, and computes exact answer keys from the final records so agent evaluation is repeatable and graded exactly. Validation tooling (realism scorecard, adversarial detector, reachability audits, and database-consistency checks) improves realism and prevents synthetic artifacts. In a reported simulator-track experiment across 23 companies, realism rose (mean score 61.8 → 97.0) and synthetic flags dropped to zero; a nine-model comparison on 33 questions yielded accuracies from 42.4% to 76.8%.
Key Points
- End-to-end generated estate:
- A single seeded generator builds an entity graph covering workforce, customers, deals, tickets, calls, messages, documents, infrastructure, etc.
- Deterministic by seed: same seed reproduces identical company and datasets.
- Product simulators:
- 66 vendor-product simulators implement each product’s API/format and project relevant slices of the shared entity graph (no simulator invents independent data).
- Projections are faithful to vendor formats and intentionally partial to mirror real systems.
- Data plane is read-only to prevent evaluation contamination.
- Internal-database generation:
- Authors supply business questions; an LLM (in the design phase) proposes schema and column-generation plans mapped to the entity graph.
- Deterministic generators materialize graph-backed records, generate database-specific facts, plant rows that satisfy/near-miss question conditions (to avoid triviality), and inject controlled data “dirt.”
- Designs are mechanically validated and can be rejected/repaired; final schemas are fixed.
- Question and answer machinery:
- Shared battery of code templates instantiates question text and computes answers directly from generated records (so answers are exact).
- Scenarios (company + questions + computed answers) are regenerated and CI-checked for byte identity to prevent drift.
- Includes single-system and composed cross-system questions; abstention/unanswerable questions are explicitly included.
- Reachability audit re-reads facts via each product API/MCP to ensure the answer is accessible to agents through real interfaces.
- Validation & realism:
- Entity-graph realism scored against targets from Eon’s operational data and public sources; adversarial detectors search for synthetic artifacts.
- Across 23 companies, mean realism increased from 61.8 to 97.0; records flagged as synthetic fell from 55.2% to 0.
- Empirical evaluation reported:
- Simulator-track comparison: nine models, same 33 questions answered three times each.
- Accuracy ranged 42.4%–76.8%; after multiple-comparison correction, 3 of 36 pairwise differences remained statistically supported.
Data & Methods
- Scenario generation:
- Inputs: four parameters (industry, company size tier, business model, application portfolio) + seed.
- Produces an entity graph (identities, employment histories, customers, pipeline, tickets, calls, documents).
- Volumes and timestamps respect realistic dependencies and correlations (e.g., larger accounts → more tickets; meetings aligned to time zones).
- Deterministic projection & serving:
- Each simulator deterministically projects the relevant slice of the entity graph in vendor-native formats; identifiers are deterministic and vendor-format valid.
- Portfolio controls which simulators are present; integrations only emit artifacts when the partner product is in the portfolio.
- Estate provisioner returns MCP endpoints and a tenant-bound credential; day states allow staged evolution (day 0 empty → day 1 full history → day 2 one more day), with answer keys recomputed per state.
- Question generation & grading:
- Templates (code) define question text and the computation that generates the correct answer from data.
- Composed cross-system questions respect join keys that are actually followable via product interfaces.
- Comparison modes include exact values, sets, sequences, mappings, and abstention; ambiguous cases for a scenario are excluded rather than arbitrarily graded.
- Internal-database design & generation:
- LLM-assisted design phase: author supplies questions; LLM proposes tables, schemas, column plans, and planting directives; validator enforces graph-backed identities and rejects incompatible designs.
- Generation phase: materialize graph-backed records, draw database-specific facts deterministically, plant satisfying and near-miss rows, add controlled dirty values (excluding graph-backed keys), and final consistency checks.
- Validation & audits:
- Database checks: plan adherence, question specificity, label integrity, dependency structure.
- Entity-graph validation: realism scorecard (against operational/public statistics) and adversarial detector for synthetic artifacts.
- Reachability audit: re-read covered facts via native APIs and MCP tools to ensure agents can retrieve the data used for grading.
- Empirical test:
- 23 generated companies used for realism tuning and evaluation metrics.
- Nine models evaluated on a simulator track with 33 repeated questions; statistical tests used to compare models and control for multiple comparisons.
Implications for AI Economics
- Measurement and comparability:
- Provides a repeatable, exact-evaluation framework for enterprise LLM agents, enabling apples-to-apples comparisons of models and agent designs—valuable for procurement decisions, benchmarking vendors, and scoring product improvements.
- Reducing data barriers and privacy risk:
- Synthetic, seed-deterministic estates obviate the need for customer production data, lowering privacy/contract barriers to development and evaluation and reducing transaction costs of accessing representative enterprise data.
- Market effects and competition:
- A standardized, extensible benchmark can accelerate competition among agent providers by making performance differences observable and reproducible, potentially lowering information asymmetries for buyers.
- Investment and adoption decisions:
- Exact, scenario-based performance metrics help quantify expected agent accuracy on enterprise tasks, informing ROI estimates, prioritization of automation projects, and risk assessments (e.g., when abstention vs. incorrect answers matters).
- Labor and task automation analysis:
- By measuring model effectiveness across real-world-like multi-system workflows (support, sales, documentation), the benchmark enables more credible estimates of task automation potential, substitutability of roles, and impacts on labor demand in customer-facing and back-office functions.
- Policy and regulatory relevance:
- Deterministic, auditable evaluation environments support compliance and auditability needs for enterprise deployments; regulators or standards bodies could use such benchmarks to set or assess minimum safety/accuracy requirements.
- Potential limitations and caveats for economic inference:
- Synthetic realism: despite high measured realism, generated estates may miss unobserved idiosyncrasies, cultural/organizational behaviors, or rare edge cases present in real firms—care is needed when extrapolating measured agent performance to live deployments.
- Coverage: benchmark results reflect the included product simulators and question templates; economic conclusions should consider portfolio heterogeneity across buyers.
- Write behavior & dynamic interaction: the benchmark is read-only; evaluation of agents that perform writes or full end-to-end process changes will need complementary testing.
- LLM-in-the-loop design bias: the internal-database design uses an LLM for schema planning; this could encode model-centric biases into tasks and influence measured outcomes.
- Recommended economic uses:
- Use as a neutral reference for vendor comparisons, procurement scoring, and cost-benefit analysis of agent deployment.
- Employ scenario sweeps to estimate heterogeneous impacts across industries, company sizes, and software portfolios.
- Combine benchmark accuracy with task-value estimates to model potential productivity gains, error costs, and net labor effects under different adoption scenarios.
Assessment
Paper Typedescriptive
Evidence Strengthn/a — This paper describes a benchmark and generation/validation methods rather than making causal claims; reported results are evaluation metrics (realism scores, model accuracies) rather than evidence for economic causation.
Methods Rigorhigh — The benchmark design is deterministic, spans many product simulators, computes answer keys from generated records, includes cross-system consistency checks, an audit that verifies reachability through product APIs, planted near-miss records to avoid trivial solutions, and iterative realism/adversarial detection; however, some validation relies on proprietary operational targets and details of adversarial detectors and scoring are not fully exposed in the excerpt.
SampleDeterministically generated fictional enterprise estates: 83 scenarios (combinations of 10 industries × 6 size tiers and 23 named companies); entity graphs containing workforce, customers, deals, tickets, calls, chats, documents; projections into simulators for 66 vendor products (e.g., Salesforce, Zendesk, Slack, Gong, S3, Google Drive); internal databases generated from author-supplied question lists; reported evaluation: nine models answering 33 templated questions three times each across scenarios; realism validation applied across 23 generated companies.
Themesproductivity human_ai_collab
GeneralizabilitySynthetic companies are calibrated to Eon’s operational targets and published sources — fidelity to other vendors' or niche industries may vary., Question templates and planted scenarios may not capture the full diversity of real-world enterprise tasks or adversarial agent behaviors., Simulators intentionally provide read-only views and may omit vendor-specific quirks or third-party integrations present in some production environments., Evaluation covered a limited set of models and 33 questions; agent performance on other tasks, workloads, or at scale may differ., Some realism and adversarial detectors are proprietary, limiting external replication and independent validation.
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The Era by Eon Benchmark deterministically generates a complete fictional enterprise from industry, company size, business model, application portfolio, and a random seed. Other | positive | Deterministic and repeatable generation of enterprise benchmark environments |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The benchmark's product simulators and internal databases describe the same fictional company through a shared underlying entity graph. Other | positive | Cross-system data consistency and joinability |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Expected answers in the benchmark are computed from the final generated records rather than manually authored. Other | positive | Exactness and reproducibility of benchmark answer keys |
Reading fidelity
high
Study strength
high
|
not reported
|
| The benchmark includes 66 product simulators, including Salesforce, HubSpot, Zendesk, Jira, Gong, Slack, Stripe, S3, and Google Drive. Other | positive | Breadth of enterprise application coverage |
Reading fidelity
high
Study strength
medium
|
n=66
66 products
|
| Across 23 generated companies, the mean realism score increased from 61.8 to 97.0. Other | positive | Synthetic enterprise data realism score |
Reading fidelity
high
Study strength
medium
|
n=23
mean realism score increased from 61.8 to 97.0
|
| Across 23 generated companies, the share of records flagged as synthetic fell from 55.2% to zero. Other | positive | Share of generated records flagged as synthetic |
Reading fidelity
high
Study strength
medium
|
n=23
decreased from 55.2% to 0%
|
| In the reported simulator-track comparison, the nine evaluated models achieved accuracy estimates ranging from 42.4% to 76.8%. Output Quality | mixed | Accuracy on enterprise-agent benchmark questions |
Reading fidelity
high
Study strength
medium
|
n=891
42.4%–76.8% accuracy
|
| After correction for multiple comparisons, only three of 36 pairwise model differences remained statistically supported. Output Quality | mixed | Statistical differences in model accuracy |
Reading fidelity
high
Study strength
medium
|
n=36
3 of 36 pairwise differences
|
| The benchmark contains 83 scenarios: one for each combination of ten industries and six size tiers, plus 23 individually named companies. Other | positive | Benchmark scenario coverage |
Reading fidelity
high
Study strength
medium
|
n=83
83 scenarios
|
| The benchmark includes twelve composed questions that require reasoning across multiple systems, and two of them test whether an agent correctly abstains when the requested information is unavailable. Decision Quality | positive | Cross-system reasoning and appropriate abstention |
Reading fidelity
high
Study strength
medium
|
n=12
12 composed questions, including 2 abstention questions
|
| The benchmark audits whether answer-key facts are reachable through both each simulator's native API and its MCP tools, and the audit has identified defects that would otherwise make questions impossible to answer through the product interface. Other | positive | Evaluability and interface reachability of benchmark questions |
Reading fidelity
high
Study strength
medium
|
not reported
|