A new benchmark of 8,000+ executable enterprise APIs finds top LLMs reach ~70% on single-hop API queries but fall to ~50% on compositional multi-hop chains and collapse on policy-constrained or unanswerable tasks, revealing language-mediated grounding—not invocation mechanics—as the primary bottleneck.

Main Finding

VAKRA is a large, tool-grounded benchmark showing that state-of-the-art LLMs still struggle with compositional, multi-hop reasoning across heterogeneous APIs and document collections under realistic tool‑use policies. Even the best model (GPT‑5.5) reaches 70.4% on simple endpoint-style (single‑hop) tasks but drops to ~50% on compositional BI APIs, degrades by over 50% as hop depth increases, and catastrophically fails on policy-constrained / unanswerable queries (as low as 2.4%). Errors concentrate in language-mediated steps (entity disambiguation, cross‑source grounding, schema/parameter alignment), not in the mechanics of invoking tools.

Key Points

• Benchmark scope

  • 8,000 executable APIs derived from real BIRD‑SQL databases across 62 domains.

  • Tasks span three difficulty tiers: (a) diverse API interaction styles (SLOT, SEL, Dashboard), (b) multi‑hop reasoning (2–5 hops) over structured APIs, and (c) multi‑source multi‑hop reasoning combining APIs and retrieval with natural‑language tool‑use policies.
  • Test set is large (≈5k queries in test split across settings).
• Evaluation approach
  • Executable, self‑hosted tool environment (Docker containers) with deterministic, re‑executable APIs + ChromaDB retrieval indices (ClapNQ and Wikidata5M).
  • Trajectory‑level verification: re‑execute predicted tool calls and compare responses (programmatic containment + LLM judge) to allow multiple valid execution paths.
  • Three‑stage waterfall evaluation: tool‑sequence verification → final response evaluation (LLM judge) → deterministic policy adherence check.
• Agent setup
  • Single, fixed ReAct harness (LangGraph/REACT) used to isolate model reasoning capabilities from agent architecture and planning hacks.
  • Tools surfaced as structured LangChain tools; agents never given task metadata (e.g., hop count).
• Dataset construction & quality
  • Multi‑hop queries generated by LLM‑assisted pipeline linking API outputs → inputs and injecting retrieval links; retrieval indices filtered to ensure source separation.
  • Human quality assessment: sampled items for multi‑hop and multi‑source settings; inter‑annotator agreement 77% and 90%; high‑quality rates 87% (multi‑hop) and 96% (multi‑source).
• Empirical findings (high level)
  • API style matters: Dashboard (endpoint) APIs easier than BI-style (SLOT/SEL). GPT‑5.5 best across styles.
  • Compositionality & depth cost: performance drops sharply with more hops; models lose >50% accuracy as depth increases.
  • Policy adherence weak: models often violate natural‑language tool‑use policies, and fail to identify unanswerable queries.
  • Failure modes localize to language tasks: entity disambiguation, cross‑source grounding, schema/parameter alignment—not tool invocation errors.
• Resources
  • Code: https://github.com/IBM/VAKRA
  • Dataset: https://huggingface.co/datasets/ibm-research/VAKRA

Data & Methods

• Tool & environment
  • Based on Elder et al. (2026) API generation pipeline; exposes >8k executable Python functions backed by SQLite/BIRD‑SQL.
  • Retrieval: domain‑specific ChromaDB indices built from ClapNQ and Wikidata5M; LLM‑based filtering applied to keep RAG/APIs separable.
  • Self‑hosted Docker image(s) that run per‑capability containers; agents interact via Model Context Protocol (MCP) over stdio.
• Task design
  • API Styles:
    • SLOT: compositional, small set of generic operations (9 tools).
    • SEL: expanded function interfaces (26 tools).
    • Dashboard: endpoint‑style, many specific endpoints (≈116 candidate tools/sample).
  • Multi‑hop reasoning: 1–5 hop chains where earlier outputs parameterize later calls (entity resolution, schema alignment required).
  • Multi‑source multi‑hop: interleaving structured API calls with unstructured document retrieval; some queries include natural‑language policies restricting allowed tools/sources.
• Query generation
  • Four‑stage LLM pipeline: extract entities, build connectivity graphs, traverse to make chains, inject retrieval passages, produce combined API+RAG queries.
  • Cross‑source answerability filtering ensures tasks require joint reasoning (can't be fully solved via only APIs or only retrieval).
• Evaluation mechanics
  • Re‑execution of predicted tool sequences against live API environment; programmatic containment checks plus GPT‑OSS‑120B LLM judge for ambiguous matches.
  • Final responses judged by GPT‑OSS‑120B (temperature 0) for groundedness and correctness.
  • Policy adherence checked deterministically for constrained tasks.
• Models evaluated
  • Frontier proprietary and open models (e.g., GPT‑5.5, Claude Opus, Gemini, Claude Sonnet, Qwen variants, GPT‑OSS‑120B, LLaMA family, etc.) under identical ReAct harness to ensure apples‑to‑apples comparisons.
• Human validation
  • Annotator study on a stratified sample across domains and clusters; rubric scored across faithfulness, logical consistency, leakiness, context sufficiency, cross‑source consistency.

Implications for AI Economics

• Enterprise deployment risk & cost
  • Compositional reasoning failures (entity disambiguation, grounding, schema alignment) imply: higher error rates in automated workflows, increased need for human oversight, and thus higher operational costs than naive claims from single‑step benchmarks suggest.
  • Policy noncompliance and inability to detect unanswerable queries create legal/compliance risks (sensitive data leaks, incorrect actions) with potential liability costs.
• Product & vendor differentiation
  • Models that appear similar on single‑hop or endpoint tasks can invert in rankings on compositional tasks—buyers should benchmark vendors on multi‑hop, multi‑source tasks, not just simple APIs.
  • There is a market opportunity for middleware and tooling (schema translators, robust entity resolvers, grounding/verification layers) that reduce language‑to‑API friction and produce audit trails; such middleware may capture significant value in enterprise pipelines.
• Procurement, pricing, and ROI
  • Procurement decisions should incorporate compositional performance, policy‑adherence metrics, and costs of necessary human‑in‑the‑loop monitoring; total cost of ownership (TCO) models must account for degradation with hop depth.
  • Vendors offering guarantees (SLAs) on multi‑step correctness or policy adherence could command premiums; conversely, buyers should price in mitigation costs when models fail at scale.
• Labor market and task automation
  • The limited reliability on multi‑hop, cross‑source reasoning suggests partial, not full, automation for complex workflows—roles that require schema mapping, data reconciliation, and policy interpretation remain resilient in the near term.
  • Investments in employee upskilling toward supervision, verification, and tooling integration will likely yield higher returns than attempts at outright substitution.
• Standardization & regulation
  • Standardized API schemas, richer metadata, and machine‑readable policy representations would reduce alignment friction and improve economic returns from automation; incentives for standardization (industry consortia, procurement requirements) could accelerate adoption.
  • Benchmarks like VAKRA provide practical evaluation primitives for regulators and auditors to assess agent behavior under compositional and policy constraints.
• Research & investment priorities
  • Economic value from improved LLMs on this front comes from targeted advances: entity disambiguation, robust cross‑source grounding, schema alignment, and formal policy‑adherence mechanisms—areas where incremental technical improvements yield outsized operational cost savings.
  • Funding/priority signals: R&D, applied engineering on grounded verification, and tool‑use policy formalization (machine‑readable policies, verifiable traces) should be prioritized for enterprise ROI.

If you want, I can: - Extract a compact checklist for enterprise procurement (metrics to require from vendors). - Produce visual figures or a slide‑ready summary of the key quantitative results (drop by hop, policy failure rates).