0 cumulative citations
View corpus contextA targeted benchmark for final contract review reveals frontier language models still fall short: the best model finds about 75% of scripted drafting defects (F1s under 0.65), highlighting limits to safely automating the last-pass legal scrub.
ContractScrub: A benchmark for final review of legal contracts
ContractScrub is a new, expert-annotated benchmark for contract 'scrubbing' showing that even frontier LLMs struggle to reliably detect drafting defects—best model reached 0.75 macro recall with F1s below 0.65 across nine error categories.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.
Summary
Main Finding
ContractScrub is a new, high-ecological-validity benchmark for the final “scrubbing” review of transactional contracts. On this task—identifying drafting defects, defined terms, and cross-reference/consistency errors across whole contracts—even frontier LLMs perform well below what their general capabilities would suggest. The best model (GPT‑5.5) attains only 0.750 macro-average recall (all F1s < 0.65), with strong results on lexically signalled categories (e.g., defined-term extraction µ≈0.835) but weak results on categories requiring contextual inference (e.g., incorrect capitalization in context µ≈0.427). This demonstrates practical limits to immediate automation of high-stakes, detail-sensitive legal work and highlights the value of narrowly targeted, domain-specific benchmarks.
Key Points
- Task definition: models must find and localize drafting defects in a full contract, returning structured tuples (category + fields). Nine categories cover defined-term extraction plus eight drafting-error types (e.g., undefined capitalized terms, unused defined terms, incorrect section/party references, inconsistent language).
- Dataset: 44 contracts (sourced from CUAD/EDGAR and curated) with 3,014 annotated instances across the nine categories. Annotations were created and validated by practicing lawyers (minimum ~8 years’ experience; many ≥10–15 years).
- Construction pipeline: (1) select & clean source contracts; (2) annotate pre-existing issues; (3) insert realistic, SME-designed errors to achieve broad category coverage; final gold answers include all instances (pre-existing + injected).
- Evaluation metric: recall-focused multiset matching per category (macro-average across categories). Recall prioritized because false negatives (missed defects) are far costlier in practice than false positives. Also report precision/F1 and a word-only variant that ignores location fields to isolate identification vs localization.
- Output normalization and scoring: lowercasing, canonicalized location fields, symmetric matching for pairwise categories; repeated occurrences scored independently.
- Models evaluated: nine frontier and open-weight models spanning families/sizes (e.g., GPT‑5.5, GPT‑5.2, o4‑mini; Claude Opus 4.7 / Sonnet 4.6 / Haiku 4.5; Qwen 3.5‑397B; Gemini 3.1 pro / 2.5 pro; Gemma‑4‑26B). Models were prompted per-category (separate instances) with structured JSON outputs parsed into tuples.
- Empirical pattern: models succeed on tasks with clear lexical cues (defined-term extraction, blatant undefined terms) but struggle on tasks requiring whole-document consistency, pragmatic intent inference, or fine-grained capitalization/context distinctions. Enabling (or using) stronger reasoning abilities yields only moderate gains concentrated on global consistency tasks rather than deeper legal interpretation.
- Practical result: even top models miss a substantial fraction of defects; none reach safe, near-human levels across all categories.
Data & Methods
- Source and size:
- Base corpus: 44 contracts drawn from CUAD / EDGAR; contracts adjusted to ~10–15 pages to be realistic and tractable.
- Gold annotations: 3,014 annotated tasks (tuples) across nine categories.
- Annotation:
- Designed and performed by 9 licensed lawyers (experience ≥8 years; many ≥10–15 years).
- SMEs annotated existing errors and also deliberately injected additional realistic errors to create a balanced test set covering the targeted error types.
- Quality control: targeted review by two additional lawyers; iterative refinements to contracts, gold answers, and prompts.
- Categories (summary):
- Defined Terms (extraction)
- Undefined Capitalized Terms
- Uncapitalized Defined Terms
- Incorrect Capitalization in Context
- Unused Defined Terms
- Terms Defined Multiple Times
- Incorrect Section/Article/Paragraph References
- Incorrect Party References
- Inconsistent Language (contradictory clauses)
- Task framing and prompting:
- Models prompted to produce a single JSON object with per-category lists; categories prompted in separate runs to reduce multitask interference.
- Scoring and normalization:
- Multiset deterministic matching of predicted tuples vs gold tuples; true positives, false positives, false negatives aggregated per category and across contracts.
- Primary metric: per-category recall (Rk) and macro-average recall across the 9 categories (Macro-R). Also report precision, F1, and a word-only variant that ignores location fields.
- Normalization: lowercasing of terms, canonicalization of location labels (e.g., section numbering, exhibits, recitals), order-independent matching for paired-location categories.
- Models evaluated: mix of proprietary frontier and open models (list above). Full hyperparameters and prompt details provided in the paper appendices.
Implications for AI Economics
- Automation potential is real but limited today:
- Contract scrubbing is economically valuable (routine, time-consuming, error-sensitive). Partial automation could reduce billable hours for tedious tasks and reallocate lawyer time to higher-value judgment work.
- Current frontier LLMs are not yet reliable enough to replace human final review; missed defects (false negatives) create legal and financial risk that limits substitution.
- Complementarity and labor reallocation:
- Likely near-term outcome: human-in-the-loop augmentation—models flag candidate issues to speed human reviewers and reduce fatigue, rather than full automation.
- This augments junior attorneys/paralegals (scaling their throughput) and changes the mix of tasks billed by firms (more supervision, fewer manual scrubbing hours).
- Investment and productization:
- High value in domain-specific fine-tuning, retrieval-augmented pipelines, or structured wrappers that improve recall/localization for this task. Firms and vendors may invest more in legal-domain adapters, chain-of-thought/consistency checks, and UI workflows that make human verification cheap.
- Benchmarks like ContractScrub can direct R&D capital to the practical gaps (full-document consistency, contextual capitalization, cross-reference resolution) rather than generic capabilities.
- Liability, regulation, and market structure:
- Because missed issues have outsized consequences, legal services buyers and regulators will demand transparency, model evaluation on domain-specific benchmarks, and contractual allocation of liability for automated outputs.
- Vendors may offer certified model-evaluation reports (using benchmarks like ContractScrub) and insurance/guarantees for enterprise usage—affecting market competition and pricing.
- Productivity vs. pricing dynamics:
- If augmentation reliably reduces time spent on scrubbing, overall productivity in transactional practices could rise. How much value is captured by firms vs. clients depends on billing models (hourly vs fixed-fee) and bargaining power.
- Increased supply of junior-attorney-level throughput could compress prices for routine contract review services, while raising demand (and margins) for high-expertise advisory and litigation services where human judgment remains crucial.
- Research & policy priorities:
- Need for more high-ecological-validity benchmarks in other professional domains to accurately forecast economic impact.
- Regulators and procurement teams should require domain-specific evaluation (recall-sensitive metrics) before permitting models to replace human review in high-stakes settings.
Limitations worth noting - Contracts are relatively short (≈10–15 pages) and come from public-company filings (CUAD/EDGAR); real-world contract distributions and drafting styles can differ. - Injected errors ensure coverage but may not reflect real-world frequency distributions. - Scoring focuses on known issues (gold annotations); unknown pre-existing errors in source contracts remain possible.
Bottom line: ContractScrub offers a practical, professionally curated way to measure LLM readiness for an economically important legal task. Current models show meaningful limitations on that task, implying near-term gains are more likely in human-augmentation workflows than in full automation; targeted engineering and domain evaluation will determine how much economic value is realized and who captures it.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic, reproducible empirical evaluation of many frontier and open models on a newly constructed, expert-annotated benchmark, giving credible evidence about model performance on contract scrubbing; however, the dataset is limited to 44 contracts drawn from CUAD/EDGAR, contains SME-inserted (synthetic) errors, and the experimental setup (e.g., prompting per-category) may not fully reflect real-world end-to-end use, reducing external validity.
Methods Rigorhigh — Annotation pipeline is carefully described and executed by experienced lawyers (multiple rounds, quality review), categories were designed by SMEs, normalization and deterministic multiset matching are specified, multiple models and metrics (recall-focused, precision, F1, word-only variant) are reported, and implementation details and prompts are provided; main methodological limitations are reliance on a relatively small, public-corpus sample, SME-inserted errors that could bias distribution of defects, and per-category prompting that may not reflect realistic single-pass usage.
Sample44 contracts drawn from the CUAD/EDGAR public corpus, annotated and edited by 9 practicing lawyers (each with ≥8 years' experience, many >10–15 years) to produce 3,014 annotated tasks across nine categories (one defined-term extraction + eight drafting-error categories); the pipeline included reviewing source contracts, annotating pre-existing issues, and purposely inserting additional realistic drafting errors; evaluated across nine frontier and open-weight LLMs (e.g., GPT-5.5, GPT-5.2, Claude variants, Qwen, Gemini, Gemma) using per-category prompts and deterministic normalized multiset matching; metrics emphasize recall and report macro-average recall, precision, and F1.
Themesproductivity human_ai_collab
GeneralizabilityContracts drawn from public US-company filings (EDGAR/CUAD) — may not represent private-commercial, international, or sector-specific drafting styles, Contracts limited to roughly 10–15 pages — longer or heavily redlined/multi-draft agreements may present different challenges, SME-inserted errors create a partially synthetic distribution of defects which may differ from naturally occurring error distributions in practice, Annotations reflect judgments of a limited set of experienced lawyers; variation in real-world reviewer standards and conventions could affect applicability, English-language, US-law-context focus — less applicable to other legal systems or languages, Evaluation used per-category prompting and JSON outputs, which may not match typical human-in-the-loop or single-pass system deployments
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| ContractScrub is the first benchmark designed to evaluate LLMs on contract scrubbing, including the identification of drafting errors and inconsistencies in full contracts. Other | positive | Benchmark coverage of contract-scrubbing capability |
Reading fidelity
high
Study strength
medium
|
n=44
|
| ContractScrub contains 3,014 annotated tasks across 44 contracts and covers nine categories of defined-term and drafting errors. Other | positive | Benchmark size and annotation coverage |
Reading fidelity
high
Study strength
medium
|
n=3014
3,014 annotated tasks across 44 contracts
|
| The best-performing evaluated model, GPT-5.5, achieved a macro-average recall of 0.750 on ContractScrub. Error Rate | negative | Macro-average recall for contract-defect identification |
Reading fidelity
high
Study strength
medium
|
n=44
0.750 macro-average recall
|
| All evaluated models had F1 scores below 0.650 on the contract-scrubbing benchmark. Error Rate | negative | F1 score for contract-defect identification |
Reading fidelity
high
Study strength
medium
|
n=44
all F1 scores below 0.650
|
| Models performed substantially better on issue categories with explicit lexical signals than on categories requiring contextual inference of intent. Error Rate | mixed | Category-level recall in contract-error detection |
Reading fidelity
high
Study strength
medium
|
n=44
defined term μ = .835; Incorrect Capitalization in Context μ = .427
|
| Current LLMs perform worse on end-to-end contract scrubbing than their performance on seemingly related general-purpose capabilities would suggest. Error Rate | negative | End-to-end contract-scrubbing performance |
Reading fidelity
high
Study strength
low
|
n=44
|
| ContractScrub is designed as a recall-sensitive task because missed contract defects are considered more costly than false-positive flags. Error Rate | negative | Relative cost of false negatives versus false positives in contract review |
Reading fidelity
high
Study strength
low
|
n=44
|
| The benchmark’s contracts were annotated and quality-reviewed by experienced lawyers, with all annotating lawyers having at least eight years of legal-practice experience. Other | positive | Annotation expertise and benchmark validity |
Reading fidelity
high
Study strength
medium
|
n=9
at least 8 years of practice experience
|