0 cumulative citations
View corpus contextEnterprise-style hardening cuts coding-agent utility: enforcing NIST-mapped access controls can reduce task success by up to 18 percentage points and inflate inference costs by as much as 167%, with different models trading success for cost in markedly different ways, meaning model choice and harness configuration materially affect suitability for hardened deployments.
Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments
arxiv
quasi_experimental
medium evidence
9/10 relevance
Full text
usable extracted full text
Source
PDF
Enforcing NIST-like runtime security controls on coding agents reduces task success and raises token costs in non-uniform ways across models and harnesses—under the strictest policy losses reach ~18 percentage points in success and cost inflation up to ~167%.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Coding agents increasingly run inside organizations whose security controls (scoped credentials, restricted egress, read-only filesystems, non-root execution) constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is unknown how performance changes when policy is enforced. In this work, we evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions. Hardening is never free but far from uniform: under the strictest policy, success losses reach 18.3 points and cost inflation 167.3\%, and the two axes disagree; the model that best preserves success is also the one that loses the most efficiency, so model choice is policy-dependent. Beyond aggregate scores, we characterize how agents behave when policy blocks their actions and decompose the failures hardening induces: runs grind into timeouts or wrong solutions rather than stopping early, in a mix that differs by model. To ground comparisons, we verify task solvability under the strictest policy, separating model failures from tasks the policy forecloses. We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents on Terminal-Bench and compatible benchmarks.
Summary
Main Finding
Security hardening materially changes coding-agent performance in non-uniform ways: enforcing realistic enterprise policies (network egress, filesystem scope, privilege) always reduces task success and increases inference cost, but the magnitude and currency of that degradation differ across model–harness bundles. Under the strictest (NIST-derived high) policy the paper records up to −18.3 percentage points in success rate and +167.3% in mean cost for some bundles. Model choice and harness design therefore become policy-dependent: the model that best preserves success can be the one that inflates cost the most.
Key Points
- Boundary-Bench: the authors release an open-source plugin (Boundary-Bench) that layers native, OS-enforced policies onto existing coding-agent benchmarks (Terminal-Bench 2.1) and measures how success and cost change as policy tightens.
- Policy ladder (three nested levels):
- Control: agent runs as root, unrestricted egress, writable filesystem.
- Non-root: drops root privilege only.
- NIST-derived high: restricted egress (allow-list proxy), read-only filesystem except a small writable workspace, and full privilege lockdown. Policies map to NIST SP 800-53 control families and are enforced at the OS level so the agent experiences them as native runtime failures.
- Experimental inventory:
- 12 frozen model–harness bundles (OpenAI GPT-5.6 variants, Anthropic Claude variants, Kimi, GLM-5.2, Qwen3.7 Max, MiniMax M3, Grok 4.5).
- Evaluated on Terminal-Bench 2.1, 89 tasks, three trials per bundle-task-policy cell.
- Primary metrics: success rate SRm(ℓ) and mean cost per task Cm(ℓ); shifts ∆SR and %∆C computed relative to control.
- Aggregate quantitative highlights:
- Every bundle loses success and gains cost under NIST-derived high (no bundle improved).
- Max observed success loss: −18.3 percentage points (Claude Sonnet 5 lost most success; GPT-5.6 Luna ≈ −18.0 pp).
- Max observed cost inflation: +167.3% (Grok 4.5 experienced the largest cost increase while losing the least success: −7.1 pp).
- Heterogeneity: some bundles trade off success for cost, others mainly pay in cost, reshaping the success–cost Pareto frontier.
- Task-level diagnostics and artifacts:
- Solvability audit under NIST-derived high: 82 of 89 tasks have a policy-compliant solvability witness (50 used original reference solution, 32 required authored adaptations); 7 tasks are blocked-by-design under the policy.
- Verifier over-specification: 5 verifiers were too strict under policy; 3 left unchanged, 2 repairs recovered additional success under hardening.
- Failure-mode decomposition:
- Hardening-induced failures are dominated by budget exhaustion (timeouts) and completed-but-wrong solutions rather than early exits.
- Agents differ behaviorally when blocked: retry loops (increasing cost), rerouting workaround trajectories, or giving up (different mixes produce different cost/success outcomes).
- Harness vs. model:
- Ablation swapping harnesses (Codex vs Grok Build) shows harness affects results but model identity still matters (Grok remains higher-cost under either harness).
Data & Methods
- Benchmark and enforcement:
- Base benchmark: Terminal-Bench 2.1 (interactive coding tasks).
- Policy enforcement: native Linux mechanisms (egress proxy, read-only mounts, unprivileged user) applied to the task sandbox; pre-flight probes verify restrictions.
- Trials run headlessly until harness returns or timeout; verifier scores final environment (timeouts can still pass).
- Evaluation design:
- 12 model–harness bundles × 89 tasks × 3 trials × 3 policy levels (control, non-root, NIST-derived high).
- Per-bundle summary metrics: success rate (percentage of passing trials over full task pool and repeats) and mean cost per task (USD).
- Shifts computed as ∆SRm(ℓ) = SRm(ℓ) − SRm(control) and ∆Cm(ℓ) = 100 · (Cm(ℓ) − Cm(control)) / Cm(control).
- Additional diagnostics:
- Solvability replay: replayed reference solutions under strictest policy; where original reference failed, authors attempted to craft policy-compliant references to establish solvability witnesses.
- Verifier audit: repaired over-specified assertions by broadening accepted workspace locations (repairs run identically at all policy levels).
- Task-level cost sampling: 100-run experiments on two representative tasks (compile-compcert, caffe-cifar-10) to inspect cost-distribution mechanisms.
- Model–harness ablation: swapped harnesses for two anchor models to disentangle harness vs model effects.
- Infrastructure and validity:
- Trials executed in isolated cloud sandboxes with task-specified CPU/memory; model inference served via vendor endpoints; invalid provisioning/harness/verifier failures were excluded and retried to ensure three valid trials per cell.
- The paper reports pooled results (no uncertainty intervals) and separates artifact-corrected outcomes where relevant.
Implications for AI Economics
- Procurement and total cost of ownership (TCO): security controls materially increase operational cost and can reduce delivered utility. Organizations must include hardening-induced inference cost inflation and reduced success rates when estimating TCO and when selecting models or service tiers.
- Model selection and vendor competition: rankings on permissive leaderboards can be misleading for enterprise deployments. Vendors that optimize for performance in hardened environments (or provide harnesses that better manage blocked actions) can obtain an economic advantage; conversely, models with better unconstrained scores may perform worse economically after hardening.
- Pricing and SLAs: observed cost inflation (up to ~+167%) implies that per-request pricing and SLA guarantees should account for environment constraints. Contracts and pricing models may need to differentiate “hardened-environment” performance.
- Product design and harness incentives: harness behavior (retry policies, routing, tool-use strategies) materially affects the success–cost trade-off under policy. There is economic value in investing engineering effort into policy-aware harnesses and routing strategies that minimize wasted inference under common enterprise policies.
- Benchmarking and procurement standards: purchasers and regulators should require policy-graded evaluation (or supply standardized hardened benchmarks mapped to controls like NIST SP 800-53) so decisions are based on realistic operational points rather than permissive sandboxes.
- Market for robustness: the heterogeneity of policy effects suggests a market niche for “policy-robust” models or subscription tiers that guarantee better performance under typical enterprise hardening—this could reshape vendor product strategies and pricing differentiation.
- Risk management: some tasks are effectively infeasible under stricter policies (7/89 blocked-by-design); security choices can therefore constrain product scope and capability. Economic trade-offs between security and functionality need explicit quantification when setting policies or investing in compensating capabilities (e.g., privileged helper services, approved tool allow-lists).
- Future economic evaluations: model value should be measured via a success–cost Pareto frontier across relevant policy levels, not a single permissive point. This affects cost-benefit analyses for adopting LLM agents, budget allocations for model usage, and investments in security-harness trade-offs.
Summary takeaway: enforcing realistic enterprise security policies changes both the success and cost of coding agents in heterogeneous ways. Evaluations, procurement, and pricing should account for this policy dependence; Boundary-Bench provides a practical tool to measure those effects.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper implements a clear experimental manipulation (policy severity) with careful environment enforcement, solvability audits, verifier repairs, and ablation checks, producing direct measurements of policy effects; limits are the bounded task set (Terminal-Bench 2.1), only 12 bundles, typically three valid trials per bundle–task–policy cell (low per-cell replication), and scarce statistical uncertainty reporting, which reduce robustness for strong causal claims across all deployments.
Methods Rigorhigh — Design enforces policies at the OS level and pre-flights restrictions, maps policies to NIST controls, audits solvability and verifier artifacts, includes ablation and targeted high-rep probes, and keeps benchmarking infrastructure consistent; remaining concerns are low per-cell replication, limited policy granularity (three aggregated levels), reliance on specific harness/vendor endpoints, and limited reporting of statistical uncertainty.
SampleEmpirical evaluation of 12 frozen model–harness bundles (OpenAI GPT-5.6 Sol/Terra/Luna via Codex, Anthropic Claude variants via Claude Code, Kimi K3, GLM-5.2, Qwen3.7 Max, MiniMax M3, Grok 4.5) on Terminal-Bench 2.1 (89 tasks). Each bundle run under three nested policy configurations (control, non-root, NIST-derived high) with three valid trials per bundle–task–policy cell; additional 100-run probes on two tasks for one bundle and a 2×2 model–harness ablation for two anchor models; execution in isolated cloud sandboxes with native Linux enforcement; solvability audit produced 82 tasks with a solvability witness and 7 tasks blocked by design.
Themesproductivity governance
IdentificationWithin-bundle before/after comparison: the authors impose three nested, OS-enforced policy levels (control, non-root, NIST-derived high) on the same set of 12 frozen model–harness bundles and 89 Terminal-Bench tasks and measure changes in task success and mean cost; they verify task solvability under the strictest policy to separate policy-blocked tasks from model failures and run ablation (model × harness swaps) and additional high-rep samples on two tasks to probe mechanisms.
GeneralizabilityResults are benchmark-specific (Terminal-Bench 2.1) and may not generalize to other software tasks or production workflows., Only 12 model–harness bundles and specific vendor endpoints were tested; other models, harnesses, or updated model versions could behave differently., Three-policy ladder aggregates multiple controls (especially the jump to NIST-derived high), so effects of individual axes (network vs filesystem vs privilege) are not fully isolated., Low per-cell replication (three valid trials) limits precision for some per-task or per-bundle inferences; transient provider-side variability could affect costs., The sandbox and enforcement mechanisms reflect one implementation; different enterprise enforcement tools or configurations could change agent behavior and cost.
Claims (12)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Under the strictest policy, every evaluated model–harness bundle experienced both lower success and higher mean cost relative to the unrestricted control. Organizational Efficiency | negative | Success rate and mean cost per task under NIST-derived high policy |
Reading fidelity
high
Study strength
medium
|
n=3204
|
| The largest observed success-rate loss under the strictest policy was 18.3 percentage points, for Claude Sonnet 5. Task Completion Time | negative | Success rate |
Reading fidelity
high
Study strength
medium
|
n=267
−18.3 pp
|
| Grok 4.5 had the smallest success-rate loss under the strictest policy, losing 7.1 percentage points, but had the largest cost inflation at 167.3%. Organizational Efficiency | mixed | Success-rate change and mean-cost inflation |
Reading fidelity
high
Study strength
medium
|
n=267
−7.1 pp success; +167.3% cost
|
| Claude Sonnet 5 incurred the largest success loss under the strictest policy while experiencing a comparatively modest 21.4% cost inflation. Organizational Efficiency | mixed | Success-rate change and mean-cost inflation |
Reading fidelity
high
Study strength
medium
|
n=267
−18.3 pp success; +21.4% cost
|
| GPT-5.6 Luna showed the smallest cost inflation under the strictest policy, at 16.0%, while losing 18.0 percentage points of success. Organizational Efficiency | mixed | Mean-cost inflation and success-rate change |
Reading fidelity
high
Study strength
medium
|
n=267
+16.0% cost; −18.0 pp success
|
| Security-policy hardening changes the success–cost Pareto frontier, and the trade-off differs across model–harness bundles. Organizational Efficiency | mixed | Joint task success and inference cost |
Reading fidelity
high
Study strength
medium
|
n=3204
|
| Grok 4.5 remained the higher-cost model than GPT-5.6 Luna under both Codex and Grok Build harnesses and under all three policy conditions. Organizational Efficiency | positive | Mean inference cost per 89-task evaluation |
Reading fidelity
high
Study strength
medium
|
n=801
|
| Crossing either endpoint model to the other harness reduced success at every policy, indicating that harness choice affected restricted-environment performance in addition to model choice. Task Allocation | negative | Task success rate |
Reading fidelity
high
Study strength
medium
|
n=801
|
| Most Terminal-Bench tasks remained solvable under the strictest policy: 82 of 89 tasks had a solvability witness, while 7 tasks were blocked by design. Task Completion Time | mixed | Task solvability under policy enforcement |
Reading fidelity
high
Study strength
high
|
n=89
82 of 89 solvable; 7 of 89 blocked by design
|
| Of the 82 tasks with a strict-policy solvability witness, 50 were solvable using Terminal-Bench's unchanged reference solution and 32 required an author-created policy-compliant reference solution. Task Allocation | mixed | Type of solvability witness under policy enforcement |
Reading fidelity
high
Study strength
high
|
n=89
50 unchanged-reference tasks; 32 author-adapted tasks
|
| Five benchmark verifiers were found to be over-specified, and adapting them increased the measured strict-policy success rate on two of those tasks. Output Quality | positive | Verifier-scored task success rate |
Reading fidelity
high
Study strength
medium
|
n=180
2 of 5 repaired tasks showed increased success
|
| For the two task-level cost case studies, hardening left GPT-5.6 Luna's success rate nearly unchanged on compile-compcert but sharply degraded success on caffe-cifar-10. Task Completion Time | mixed | Task success rate under policy hardening |
Reading fidelity
high
Study strength
medium
|
n=400
|