A misconfigured serving adapter can make a model's tool calls vanish — the same checkpoint reports 0% tool use under one parser/template and near-full use under another — and the silent censorship scales with model size and seeps into RL training, removing tool-mediated trajectories from the learning signal.

Main Finding

The observed tool-call rate (and therefore reported tool-use capability) is not a property of the model alone but of the composed model–interface measurement stack (model × prompt/template × serialization/envelope × parser × execution). A mismatched serving interface can entirely censor tool-intended trajectories — reporting zero parsed calls while the model emits well-formed calls — and this censoring is scale-dependent and directly affects RL training (the policy never sees tool-mediated experiences if the interface discards them).

Key Points

• Interface-induced trajectory censoring: when the serving/adapter stack fails to accept the model’s emitted format, the system reports HTTP 200 and an empty tool_calls array — indistinguishable at the output from a model that never tries to use tools.
• Two failure modes:
  • Masking: a valid call is not recognised by the server and never reaches execution.
  • Suppression: an invalid or out-of-contract emission is removed before it could be an action sample.
• Pure interaction effect (example): on BFCL v4, holding weights, seeds, executor and scorer fixed, changing only parser + chat template moves the same model’s reported score from 0.00 to 0.96 / 0.19. A 2×2 test shows both main effects ≈ 0 and the entire effect in their interaction — neither model nor single component is “buggy” alone.
• Large-scale examples:
  • BFCL v4: documented adapter → 0/200 parsed; dedicated adapter → 196/200 parsed; repaired arm executes and rescues items the documented arm never did.
  • τ-bench (115 retail tasks): parsed calls 0 vs 636; tasks reaching any tool execution 0 vs 103 when only the adapter changes.
  • Qwen2.5-Coder scale ladder (1.5B → 32B, n=100 per size): server-parsed calls = 0 at all sizes under a documented hermes configuration, while emitted well-formed calls rise to 80/100 at 32B (≈72 after adjudicated correction). Under a matched envelope (preregistered counterfactual), parsed counts are nonzero (e.g., 86/44/70/97 at 0.6–8B in a similar ladder) and the “silent fraction” falls to ~0–2.
  • Llama-3.1-8B: a 23% role-confusion rate (calling the task function as a tool) is eliminated under strict: true.
• RL training effect: in verl’s AgentLoop at 7B, 45 of 115 generations include a complete call but 0 are accepted/executed/return observations — so the sampled experience distribution contains no tool-mediated trajectories to reinforce; fixing the adapter at evaluation restores parsing but does not always yield statistically significant outcome gains.
• Validation and controls:
  • Replaying vLLM’s extractor over stored bytes finds zero cases where the server failed to parse a call its extractor would accept — the failures are about contract mismatch, not a parser bug.
  • The authors preregistered the matched-envelope counterfactual before running and release a 98-line preflight check that catches every silent failure reported.
• Measurement warning: reporting only server-parsed tool_calls (common practice) confounds model capability with serving/interface choices and can undercount or totally hide tool use, with the bias growing with model scale in observed cases.

Data & Methods

• Benchmarks and tasks:
  • BFCL v4 (official data, executor, scorer), 200 cases for the demonstrated run.
  • τ-bench (115 interactive retail tasks; stateful simulated user environment).
  • Custom probe on 100 problems from a decontaminated KodCode subset (paired arms).
• Protocols compared:
  • ReAct (text protocol: Thought / Action / Action Input / Observation).
  • OpenAI-style function calling (FC) variants: terse, rich, strict (strict: true, additionalProperties: false).
  • Serving flag: tool_choice: auto vs required.
• Models and serving:
  • vLLM 0.27.1 serving stack, family-specific parsers/templates.
  • Qwen2.5-Coder ladder at 1.5B, 3B, 7B, 14B, 32B; Qwen3 ladder for matched-envelope counterfactual; Llama-3.1-8B example.
• RL training:
  • GRPO and RLOO used for agent training; verl AgentLoop inspected to measure whether tool calls become executed actions in rollouts.
• Emitted-intent classifier:
  • Single classifier with nested tiers (tight, strong, weak) validated against third-party adjudicated gold standard.
  • Classifier inter-rater performance κ ≈ 0.871; tight-tier precision 36/40 = 0.90 used to calibrate reported emitted-call counts.
• Diagnostics:
  • 2×2 template × parser experiments to isolate interaction effects.
  • Re-parsing stored bytes under multiple rules and replaying vLLM extractor line-by-line to rule out parser bugs.
  • Preflight check (98-line script) released with code/data/preregistration at the project repo.
• Availability:
  • Code, data, and preregistered predictions available: https://github.com/nebula-1999/Interface-Induced-Trajectory-Censoring

Implications for AI Economics

• Measurement and valuation distortions
  • Benchmarks drive research priorities, procurement, and product positioning. If reported tool-use metrics depend on serving adapters and templates, then comparative model valuations (e.g., “Model A has X% tool-use” vs “Model B has Y%”) can be misleading. Buyers and researchers may under- or over-pay for capabilities that are actually contingent on integration choices.
• Misleading market signals and misallocated R&D
  • Apparent scale-dependent changes in tool-call rates (larger checkpoints emitting more well-formed calls but being undercounted by mismatched adapters) could cause incorrect inferences about the returns-to-scale for tool-use capability. This affects decisions about whether to invest in larger models or instead invest in better adapters and tooling.
• Hidden compute and opportunity cost in RL training
  • When the interface censors all tool-mediated trajectories, RL training does not receive gradient signal for tool use and can waste compute (and time) trying to learn behaviors it cannot observe in rollout rewards. Institutions may need to account for increased sample inefficiency and failed experiments in cost estimates.
• Benchmarking, procurement, and contracting practices
  • Contracts and procurement decisions that rely on reported benchmark numbers should require disclosure of serving/adapter configurations and validation that measured behaviors reflect the model’s emissions (e.g., running a preflight check). Otherwise, institutions risk procuring models whose reported capabilities were instrument-dependent.
• Standards, certification, and regulation
  • Regulators or industry bodies creating model capability standards (for safety or performance) should specify measurement stacks and require reproducible interface configurations. A minimal standard could mandate preflight checks and disclosure of parsers/templates used to produce reported metrics.
• Incentive alignment for adapter/stack development
  • The findings reframe a portion of capability improvements as engineering-on-the-stack rather than pure model improvements. This increases the economic value of adapter tooling and parser design; firms specializing in serving stacks may capture disproportionate returns if they can reliably recover censored capabilities.
• Auditability and reproducibility costs
  • Auditors must verify not only model weights and prompts but also serving adapters and parser contracts. This increases the scope and cost of independent benchmarking and audits; active disclosure and automated preflight tests (like the provided 98-line check) reduce those costs.
• Risk of misreporting in leaderboards and research comparisons
  • Leaderboards and published evaluations that do not fix or report serving adapters can present incomparable scores. Economists and policymakers interpreting progress in tool use must account for measurement risk; naive time-series of “capability growth” may partly reflect changes in interface choices.
• Practical managerial recommendations (economically relevant)
  • Require standardized measurement interfaces (or report the full stack) for procurement and research comparisons.
  • Include interface validation (e.g., the paper’s preflight check) in any benchmark pipeline to avoid hidden zeroes.
  • Treat adapter engineering as a first-order investment: low-cost adapter fixes can unlock capabilities without retraining models.
  • When budgeting RL experiments, add contingency for adapter-induced censoring (diagnostic/validation time and potential retraining).
• Limitations to economic generalization
  • The paper documents a concrete set of families, serving stack (vLLM), and models (Qwen ladder, Llama-3.1). Generality across all stacks and domains is plausible but should be empirically validated; some domain-specific payload failure modes differ. Still, the core observation — that measurement depends on the entire pipeline — is robust and economically consequential.

If you want, I can: - Extract a short checklist (operational preflight) for procurement/audit teams based on the paper’s 98-line check. - Produce a short note for procurement contracts specifying required interface disclosures and tests.