An AI-in-the-loop tool for UK planning departments automates redaction and metadata extraction and is being piloted across four councils, promising to cut administrative workload and legal risk—but evidence to date is limited to pilot measurements and modelled savings.

Main Finding

An AI-in-the-Loop (AI2L) Intelligent Document Processing system for UK local planning authorities can substantially reduce manual administrative effort and compliance risk in processing planning application documents. In controlled PoC results, the system delivered high PII recall and improved extraction/detection accuracy versus classical baselines, halved median processing time per document, and produced high user acceptance. The system is being piloted at four diverse UK authorities; deployment emphasizes human confirmation, auditability, and incremental learning rather than autonomous approval.

Key Points

• Problem: UK planners must publish planning documents (Town & Country Planning Act) while protecting personal data (Data Protection Act 2018), creating high-volume, high-risk redaction and metadata tasks that are manually intensive.
• Design choice: AI-in-the-Loop (AI2L) — the system suggests extractions, PII detections, and visual detections but requires explicit human confirmation before any commit (no automated approval).
• Workflows:
  • Data extraction and verification: pre-populate metadata (Title, Date, Scale) for planner review.
  • PII detection & redaction review: list candidate PII with confidence scores; planner confirms each redaction.
  • Visual checks: detect elements on drawings (e.g., North arrow, scale bar) with overlays and deterministic validation rules; planner confirms.
• Integration & safety: enterprise integration with planner software, immutable audit logs, flattened (true) redaction of content, post-commit scrub to prevent recoverable residuals, configurable pre/post-processing and fallback prompts.
• Active learning: the system prioritizes uncertain cases for human review to improve models over time but does not self-approve.
• Pilot: deployed at four authorities representing different contexts (apps/year ~6,000; 4,500; 5,000; 2,000).

Data & Methods

• Development path: rapid PoC (VLM-based, Streamlit demo) → integrated pilot (secure backend, enterprise UI, configurable hybrid pipeline).
• Dataset for controlled evaluation: 440 planning-application documents (diverse types) from two authorities; ground-truth mostly derived from previous planner-processed records (annotated by a single planner per label in most cases).
• Metrics (PoC / controlled test set):
  • NER F1: Title 0.84 (vs baseline 0.76), Date 0.88 (vs baseline 0.81).
  • Object detection mAP@.5 IoU: North Point 0.78 (vs baseline 0.55), Red Line 0.71 (vs baseline 0.42).
  • PII recall (per-category): Names 0.97 (baseline 0.90), Addresses 0.94 (0.85), Emails 0.99, Phones 0.98, Signatures 0.92. Micro/macro recall 0.96/0.96.
  • Operational/UX: median time per document 35s (baseline 65s), suggestion acceptance rate 92%, System Usability Scale 78.5.
• Cost accounting (PoC token-based): median per-document VLM cost ~ $0.046. Provider-range illustrative medians: Google $0.046, OpenAI $0.042, Anthropic $0.052; two-pass fallback path ~ $0.071/doc. Costs exclude full integration, hosting, and personnel overheads.
• Baseline systems: OCR + CRF NER + regex/dictionary for PII + traditional CV for symbols; used for comparative evaluation.
• ROI tool: interactive web model where authorities input local variables (apps/year, docs/app, officer fully-loaded hourly rate) to estimate hours saved, FTE unlocked, net financial benefit, and payback period. Used as a deployment conversation and buy-in tool (not a definitive predictor).

Implications for AI Economics

• Productivity vs. Augmentation: The system demonstrates measurable productivity gains (median per-doc time roughly halved in PoC). Because the design augments rather than replaces planners, gains are realized as capacity freed for higher-value work rather than direct headcount reductions—important for public-sector budget and labor negotiations.
• Labour-cost ROI: Low per-document model costs (~$0.04–0.07) imply that even modest time savings per document can yield positive ROI for authorities with thousands of annual applications. The ROI tool helps translate technical performance into financial terms tailored to each authority.
• Scaling and marginal economics: VLM token costs scale linearly with document volume; provider choice, model tier, and fallback strategies materially affect marginal cost per doc. Integration and operational fixed costs (engineering, hosting, secure endpoints, audit/logging) will dominate for smaller authorities; consortium or shared-service models (e.g., multi-authority deployments) can improve unit economics.
• Risk reduction value: High PII recall and safe redaction procedures reduce legal and reputational risk from improper disclosures. Quantifying avoided compliance costs and potential fines can materially change the economic case, especially in risk-averse public institutions.
• Active learning & label efficiency: Prioritizing uncertain cases for human review reduces annotation cost and focuses human effort where it delivers the largest model improvement. Over time, reduced manual verification rates can lower per-document human time, improving long-run economics without sacrificing human accountability.
• Procurement and governance costs: Public-sector deployments face non-negligible governance, audit, and procurement overhead (security reviews, vendor risk assessment, data protection impact assessments). These fixed transactional costs should be included in cost-benefit analyses and favor solutions that provide strong auditability and demonstrable data governance (as this system does).
• Distributional effects & externalities: By increasing transparency and standardization of metadata extraction, the system may lower transaction costs and improve downstream data platforms (e.g., Planning Data Platform), generating public-good benefits. Conversely, reliance on external VLM providers creates vendor lock-in and privacy/extraterritoriality considerations that carry economic and policy externalities.
• Limitations & uncertainties impacting economic forecasts:
  • Controlled PoC metrics may not fully generalize; pilot is ongoing and real-world error modes, annotation variability, and UI/behavioral factors may change realized savings.
  • The dataset annotations had single-planner ground-truth for many labels; label noise or inter-annotator variance can affect measured accuracy and thus operational confidence.
  • Reported per-document VLM costs exclude integration/maintenance and governance overhead; full TCO should be computed per-authority.
  • Legal/regulatory changes or stricter interpretation of disclosure obligations could increase human confirmation burdens, altering the economic case.

Overall, the paper presents an applied example where modest per-document AI costs, combined with significant time savings and compliance risk reduction, can yield favorable economics for large-volume public-sector document workflows. The economic viability is sensitive to scale, integration fixed costs, provider pricing, and realized model accuracy in production; tools like the described ROI calculator are valuable for translating technical performance into locally-relevant economic projections.