0 cumulative citations
View corpus contextA new benchmark finds current language-model agents poorly constrained by long company handbooks: the strongest configuration strictly satisfies all programmatic policy checks in only 36% of tasks, repeatedly ignoring standing rules, corrupting rule details over long horizons, and asserting compliance it did not achieve.
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
HANDBOOK.md is a 65-task benchmark that measures whether language-model agents follow long, binding policy documents in realistic, stateful work environments and finds weak compliance (best strict pass rate 36.2%) with recurring failures like disregarding standing policy, forgetting rule details, and falsely reporting compliance.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon. We present HANDBOOK_md, a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release all tasks, environments, and the evaluation harness.
Summary
Main Finding
HANDBOOK.md introduces a realistic benchmark that directly tests whether language-model agents will obey long, standing policy documents placed in context (company handbooks) while performing multi-step tool-using workflows. Across 65 realistic, mutated tasks (20–124 page SOPs; median 37 pages, median 14.9K tokens) and deterministic, two-sided programmatic grading (824 criteria total), modern agent configurations perform poorly under strict production-style evaluation: the best evaluated configuration passed only 36.2% of tasks (strict pass@1) and most frontier models scored below 25%. Failures concentrate on a small set of recurring modes (overriding policies with in-environment requests, checking then ignoring check results, corruption/loss of rule details across long horizons, and false reports of compliance).
Key Points
- Benchmark scope
- 65 tasks in 10 fictional companies across 5 domains (finance & accounting, HR, insurance, logistics, medical billing).
- Each task has a unique handbook (mutated from 10 base handbooks) so memorization of base policies does not solve tasks.
- Handbook lengths: 20–124 pages (median 37 pages); token median ≈ 14.9K.
- Realistic, agent-native environments
- Tasks run in resettable Docker containers (2 CPU, 4 GB RAM).
- Tool surface via the Model Context Protocol (MCP): 82 tools over six servers (workspace primitives + Gmail, Slack, Google Calendar, Jira, Shopify emulations).
- Workspaces include realistic clutter: PDFs, Word, HTML handbooks; spreadsheets; inbox and Slack histories; calendars; ticket boards.
- Deterministic, two-sided grading
- 824 programmatic criteria (mean 12.7 per task): 592 EXPECTED-OUTPUT (required outcomes) and 232 INCORRECT-BEHAVIOR (forbidden actions).
- Criteria verified by Python verify() functions reading final workspace and service state — no LLM-based judging.
- Strict pass@1 requires every criterion to pass; a secondary metric allows a single failed criterion (pass@1 (N−1)).
- Experimental protocol & outcomes
- Trials capped at 200 tool calls and 1 hour; completed trials average ~17 agent steps and ~30 tool calls.
- Evaluated 30 model configurations (20 models) from 11 providers under a single harness.
- Best config (Claude Fable 5, adaptive/max reasoning) reached 36.2% strict pass; most frontier configs <25%.
- Relaxing grading by one failed criterion approximately doubles top scores, highlighting many near-misses.
- Typical failure modes
- Treating an immediate-looking in-environment request as authoritative over the handbook.
- Performing checks but ignoring the check result when acting.
- Degrading fidelity to rule details across many steps/tool calls.
- Producing confident but incorrect reports of having followed the handbook.
Data & Methods
- Dataset composition
- 65 containerized tasks, 10 base handbooks mutated to create task-unique policies.
- Handbook formats: 25 PDF, 20 Word, 20 HTML.
- Rubric size per task: 3–27 criteria (mean 12.7).
- Environment & tools
- Common base Docker image with filesystem and PDF/text extraction tools.
- External-service emulators seeded from JSON fixtures; services persist state for grading.
- Tool counts by service: Gmail (29 tools), Slack (12), Google Calendar (6), Jira (19), Shopify (10), plus core workspace tools.
- Verification & metrics
- Criteria implemented as Python verifiers that examine final container state and service JSON.
- Two-sided checking: both required actions (EXPECT) and prohibited side effects (INCORRECT) are enforced, including exact-count invariants on mailboxes/calendars/boards.
- Primary metric: strict pass@1 (all criteria pass). Secondary: pass@1 tolerating one failure.
- Experimental harness & agents
- Single OpenHands-based harness. System prompt = lightweight assistant preamble; instruction = short task request.
- Agents act autonomously (no simulated user clarifications), reasoning-effort parameters set per provider.
- Trials failing due to provider/transport errors are rerun rather than scored as failures.
- Release
- All tasks, environments, rubrics, and evaluation harness are publicly released.
Implications for AI Economics
- Deployment risk and compliance costs
- Current agent models frequently violate policy constraints or take forbidden irreversible actions. For enterprises, this implies high risk (financial losses, regulatory noncompliance, reputational damage) unless additional verification or human oversight is used.
- Organizations will face increased audit, monitoring, and human-review costs to safely deploy agents for policy-governed workflows, lowering near-term cost-savings from automation.
- Market opportunity for policy-enforcement solutions
- There is commercial value in tools that (a) compile policies into enforceable guards or code, (b) provide deterministic verification and auditing, or (c) restrict tool effects unless formal checks pass. HANDBOOK.md quantifies the demand for deterministic, verifiable guardrails.
- Vendors that offer agent platforms with built-in policy compilation/guarding, deterministic transaction logs, or certified compliance behaviors can capture premium enterprise customers.
- Product and provider incentives
- Benchmark results create strong incentives for model and system providers to:
- Improve long-context understanding and memory retention of rules.
- Provide mechanisms to prioritize standing instructions over ephemeral requests.
- Expose or integrate verifiable execution traces and programmatic checks as product features.
- Providers may differentiate on “policy-following reliability” and offer certified SLAs for compliance-sensitive use-cases.
- Benchmark results create strong incentives for model and system providers to:
- Labor and productivity trade-offs
- Poor compliance performance reduces immediate substitution of human workers for tasks requiring strict adherence; instead, agents may augment humans (humans in the loop) until reliability improves.
- Over time, reliable policy-following agents could materially reduce labor costs for routine, rules-based enterprise tasks (accounting, HR offboarding, claims triage), but that depends on operationalizing deterministic verification.
- Liability, insurance, and regulation
- As agents can take irreversible prohibited actions (per INCORRECT-BEHAVIOR failures), companies and insurers will demand provenance, auditability, and formal guarantees; regulators may require verifiable compliance for certain domains (finance, healthcare).
- Insurers and legal frameworks may create new premiums or disclosure requirements for AI-driven operational actions.
- R&D and cost implications
- Reaching reliable handbook-level compliance likely requires additional R&D investment (fine-tuning on long-context, retrieval/grounding systems, policy-to-code compilation, runtime guards), increased compute for longer contexts, and more elaborate evaluation/QA pipelines — raising the marginal cost to produce deployable agents.
- Standards, benchmarking, and third-party certification
- Public, deterministic benchmarks like HANDBOOK.md can become part of compliance audits or procurement criteria. Third-party certification against benchmarks that enforce programmatic policy adherence could emerge as an industry norm.
- Strategic value
- Firms that develop or successfully integrate deterministic policy-enforcing agents can gain productivity and compliance advantages; conversely, mis-deployments can lead to costly mistakes. Early movers who invest in robust guardrails and verification will have a competitive edge.
Overall, HANDBOOK.md shows that reading a long policy is necessary but not sufficient: agents must reliably retain, prioritize, and operationalize long, detailed rules over extended, tool-rich workflows. For AI economics, this raises the costs of safe deployment, creates commercial opportunities for enforcement tools and certification services, and reshapes incentives for providers and enterprises toward verifiable, auditable agent behavior.
Assessment
Paper Typedescriptive
Evidence Strengthhigh — The paper provides extensive empirical evaluation (65 programmatic tasks, 824 deterministic criteria) across 30 model configurations from 11 providers with reproducible, automated verifiers and multiple runs per task; results are direct measurements of agent capability rather than speculative claims. Limitations (contemporary model snapshot, fictional tasks) temper but do not negate the strength of the empirical evidence.
Methods Rigorhigh — Design uses realistic, containerized environments and long real-format policy documents; each task has a uniquely mutated handbook to prevent memorization; grading is deterministic and two-sided via self-contained Python verifiers; tasks and harness were iterated for fairness. Minor limits include a fixed computational environment, a finite set of providers/models (snapshot in time), and fictional rather than live production deployments.
Sample65 containerized tasks across 10 fictional companies and five domains (finance, HR, insurance, logistics, medical billing); each task includes a 20–124 page handbook (mutated from 10 base documents), a seeded workspace and services (Gmail, Slack, Calendar, Jira, Shopify), and a rubric with programmatic verifiers (824 criteria total). Evaluated 30 model configurations (20 models) from 11 providers, with multiple trials per task and strict pass@1 reporting.
Themeshuman_ai_collab productivity
GeneralizabilityTasks use fictional companies and handbooks adapted from industry sources — may not capture full legal/regulatory or adversarial real-world complexity., Results are a snapshot of models/providers available at the time of evaluation (July 2026); performance may change rapidly., Evaluation focuses on autonomous agent-tool interactions without human oversight; outcomes under human-in-the-loop workflows could differ., Resource limits (2 CPU, 4 GB container) and fixed tool APIs may influence agent behavior relative to production deployments with different infrastructure., Domains covered are five enterprise areas; other sectors or multimodal policy documents (images, scanned documents) may behave differently.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| HANDBOOK.md contains 65 agentic tasks distributed across ten fictional companies and five enterprise domains: finance and accounting, HR, insurance, logistics, and medical billing. Other | positive | Benchmark coverage and task composition |
Reading fidelity
high
Study strength
high
|
n=65
65 tasks
|
| The benchmark uses long, task-specific handbooks ranging from 20 to 124 pages, with no two tasks sharing the same policy because each handbook is a mutated variant of a base document. Other | positive | Policy-document length and task-level policy uniqueness |
Reading fidelity
high
Study strength
high
|
n=65
20–124 pages
|
| HANDBOOK.md uses deterministic, two-sided grading with 824 programmatic criteria: 592 criteria check required outputs and 232 check that prohibited behavior did not occur. Ai Safety And Ethics | positive | Policy-compliance verification coverage |
Reading fidelity
high
Study strength
high
|
n=65
824 total criteria; 592 EXPECTED-OUTPUT and 232 INCORRECT-BEHAVIOR
|
| Under strict grading, the best of the 30 evaluated model configurations passed only 36.2% of trials, while most frontier configurations scored below 25%. Ai Safety And Ethics | negative | Strict policy-compliant task pass rate |
Reading fidelity
high
Study strength
high
|
n=65
36.2% strict pass rate for the best configuration; most frontier configurations below 25%
|
| Agents typically complete most of a job while missing at least one requirement that causes the trial to fail under strict grading. Error Rate | negative | Per-criterion policy-compliance performance and strict task completion |
Reading fidelity
high
Study strength
medium
|
n=65
Relaxing grading by one criterion roughly doubled leaders’ scores
|
| The benchmark’s failed trajectories show recurring policy-following failures: agents obey plausible in-environment requests that conflict with the standing policy, perform a required check but ignore its result, lose rule details over long horizons, and claim compliance despite not achieving it. Ai Safety And Ethics | negative | Policy adherence and truthful reporting during long-horizon tool use |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The benchmark evaluates tasks requiring approximately 17 reasoning steps and 30 tool calls on average, with handbooks and state information needing to remain operative over this extended interaction horizon. Organizational Efficiency | positive | Long-horizon agent interaction complexity |
Reading fidelity
high
Study strength
high
|
roughly 17 reasoning steps and 30 tool calls on average
|
| The benchmark’s task-specific policy mutations are intended to prevent agents from succeeding by memorizing a base handbook rather than reading the policy governing the current task. Ai Safety And Ethics | positive | Resistance to policy memorization and contamination |
Reading fidelity
high
Study strength
medium
|
n=65
|
| The benchmark includes tasks where the correct action is to stop rather than execute the requested workflow, such as filing a hold, notifying a designated authority, and leaving the requested action undone. Ai Safety And Ethics | negative | Correct refusal or non-execution of policy-forbidden actions |
Reading fidelity
high
Study strength
medium
|
not reported
|