A new benchmark for Microsoft’s Business Central DSL finds large disparities in coding-agent performance and shows that gains on general-purpose code tests often fail to translate to ERP-specific engineering tasks; model choice matters more than the agent harness for bug fixes in the evaluated settings.

Main Finding

Agentic coding systems perform substantially worse on a domain-specific ERP language (AL for Microsoft Dynamics 365 Business Central) than general-purpose benchmarks imply. In BC-Bench (101 real-world AL tasks), differences across underlying LMs explain more variation in bug-fixing success than differences across two widely used agent harnesses (GitHub Copilot and Claude Code). Improvements reported on general-purpose code benchmarks do not reliably transfer to AL; task complexity (multi-file or larger patches) and domain constraints substantially reduce agent accuracy.

Key Points

• Benchmark introduced: BC-Bench — 101 manually curated, real-world tasks drawn from two Microsoft repositories (NAV, private; BCApps, public). Open source: https://github.com/microsoft/BC-Bench
• Two evaluation categories:
  • Bug Fixing: generate a patch that builds and makes tests pass.
  • Test Generation: generate a test that fails on the base commit and passes after the gold patch.
• Agent harnesses evaluated: GitHub Copilot and Claude Code. Multiple frontier models evaluated (e.g., claude-opus-4.6, claude-opus-4.5, GPT-5.x-codex variants, GPT-4.1).
• Metrics and evaluation protocol:
  • Five independent runs per task to capture stochasticity.
  • Mean resolution rate (with bootstrapped 95% CI across runs).
  • passˆ5 (fraction of tasks solved in all five runs) to quantify reliability.
  • Wall-clock duration per trial reported.
• Representative results (Bug Fixing, matched settings):
  • claude-opus-4.6 (Claude Code): mean resolution 68.5% (95% CI 65.7–71.3%), passˆ5 = 49.5%, duration 284s.
  • claude-opus-4.6 (GitHub Copilot): mean 65.1%, passˆ5 = 50.5%, duration 314s.
  • GPT-5.2-codex (GitHub Copilot): mean 60.8%, passˆ5 = 49.5%, duration 196s.
  • GPT-4.1 (GitHub Copilot): mean 16.6%, passˆ5 = 5.0%.
• Model vs harness effects:
  • Matched comparisons indicate model choice drives statistically significant differences (examples: claude-opus-4.6 vs 4.5, p = 0.026; gpt-5.2-codex vs gpt-5.1-codex-max, p = 0.019).
  • Harness comparisons (Claude Code vs GitHub Copilot with same model) did not reach statistical significance in tested pairs (p > 0.05), suggesting smaller harness effect in the evaluated Bug Fixing setting.
• Task complexity strongly impacts performance:
  • Multi-file patches: mean resolution drops >20 percentage points compared to single-file fixes (e.g., ~65–70% → ~42–43%).
  • LoC changed: 1–10 LoC tasks ~75–78% success; 11–25 LoC drops to ~38–50%.
• Domain heterogeneity: performance varies by functional area (e.g., Inventory and Sales higher; Finance lower for some models).
• Multimodal support: 67 of 101 tasks include images/screenshots in problem statements; BC-Bench supports multimodal problem statements.
• Environment and reproducibility:
  • Tests executed in GitHub Actions on standardized containers (BcContainer-Helper).
  • Agents by default see code as text only; an optional tooling interface (AL MCP) can expose compilation to agents.
  • Versioning strategy and weekly revalidation to ensure reproducibility; timeouts (30 → 60 minutes) and workflow-failure handling described.

Data & Methods

• Dataset construction pipeline (five stages):
• Manual repository selection (two Microsoft-managed repos).
• Attribute filtering: single-bug-fix PRs that introduce at least one test.
• Manual filtering to ensure sufficient context and remove brittle tests.
• Bug-PR pair extraction with AL-specific adaptations: multi-project mapping, release-based anchoring, base-commit alignment; gold patches are the human PR fixes.
• Execution filtering: automated test runs to ensure failing-before/ passing-after properties; re-run weekly to detect regressions.
• Dataset characteristics:
  • Total tasks: 101 (85 BaseApp, 16 other apps).
  • Avg files modified (gold patch): 1.3; median 1.0.
  • Avg gold patch LOC: 18.9; median 9.0.
  • Tasks with images: 67.
  • Functional area distribution: Inventory (21), Finance (19), Sales (12), etc.
• Evaluation environment and protocol:
  • Five independent runs per task; mean resolution and bootstrapped 95% CI (BCa with 10,000 resamples).
  • passˆ5 reported to measure reproducibility across runs.
  • Wall-clock duration measured (model calls and tool uses).
  • Statistical testing: two-sided exact sign-flip permutation test on paired per-task success-count differences.
  • Execution via GitHub Actions using standardized self-hosted runners and Docker containers mapped to release versions.
• Limitations in data:
  • Small DSL-specific corpus relative to general languages (101 tasks), largely from Microsoft-owned repos; public AL code is scarce.
  • BC-Bench focuses on tasks that have machine-executable tests and meet filtering criteria; not a full coverage of all AL engineering work.

Implications for AI Economics

• Need for domain-specific evaluation before adoption:
  • Firms should not rely solely on general-purpose benchmark gains to justify investment in agentic engineering for DSL-heavy domains. BC-Bench shows transfer is imperfect; empirical validation in the target domain is necessary.
• Productivity and ROI are task-dependent:
  • High automation potential for small, single-file changes (1–10 LoC) where models reach ~75–78% mean resolution. These are low-hanging fruit for cost-saving and developer speedups.
  • Complex, multi-file, or larger patches remain hard for current models (large drops in success), implying limited ROI for fully automating complex ERP maintenance without further tooling or human-in-the-loop workflows.
• Model selection matters more than harness choice (in evaluated settings):
  • Organizations should prioritize model evaluation and selection for domain-specific tasks; switching harnesses (Copilot vs Claude Code) had smaller impact than upgrading underlying models in the BC-Bench Bug Fixing experiments.
• Latency vs accuracy trade-offs affect economic value:
  • More accurate models (e.g., claude-opus-4.6) can be substantially slower. Cost-benefit analyses must incorporate runtime latency and associated operational costs (compute, developer waiting time).
• Investment priorities to improve agent performance in DSL contexts:
  • Tooling integration (e.g., exposing compilation/runtime via AL MCP) could materially improve results — investing in agent-accessible development infrastructure can raise automation rates.
  • Curating domain-specific datasets and benchmarks (though costly) is necessary for realistic evaluation and targeted model improvement.
  • Multimodal and environment-aware agents: ERP bug reports often include screenshots and require environment coupling; enabling agents to use those signals can increase effectiveness.
• Market and policy considerations:
  • Limited public DSL corpora (proprietary industrial code) mean vendors or platform owners may hold asymmetric advantages; collaborations or sanctioned dataset releases could accelerate progress but raise IP and privacy questions.
  • Reproducibility/versioning requirements imply ongoing evaluation costs — firms should budget continuous benchmarking and re-validation as models and harnesses evolve rapidly.
• Strategy recommendations for firms:
  • Pilot automation on well-scoped, high-frequency, small-patch tasks to realize early gains.
  • Evaluate multiple models on a representative domain-specific benchmark (like BC-Bench) rather than relying on public general benchmarks.
  • Invest in tooling that exposes compilation/tests to agents to bridge the gap between text-only prompts and executable domain workflows.
  • Track latency, passˆk-like reliability metrics, and operational workflow failure modes (timeouts, non-submissions) when estimating realized productivity improvements.

If you want, I can extract a compact table of the main numeric results (mean resolution, passˆ5, durations) for the top models/harness combinations, or draft a short checklist for a firm considering deploying agentic engineering in an ERP/DSL setting.