MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loop

Document processing automation remains a critical challenge in enterprise environments, where traditional manual approaches are labor-intensive and error-prone. We present MADP, a multi-agent architecture that addresses the challenge of automating document processing in enterprise settings by combining deep learning-based classification and parsing with large language model extraction, while maintaining accuracy through selective human validation. Our system integrates five specialized agents--Classificator, Splitter, Parser, Extraction, and Validator--with a Human-in-the-Loop (HITL) mechanism and a novel Prompt Fine Tuning with Feedback Inheritance (PFTFI) approach. The operational analysis on a production use-case scenario of 100,000 invoices per year indicates a potential reduction of Full-Time Equivalent (FTE) requirements by approximately 70%. Production deployment on 955 real-world documents processed through January 2026 achieves a 97.0% full-pipeline automation rate, with only 3% requiring non-AI fallback. Ablation evaluation on a stratified 100-document subset (5 documents per each of 20 supplier/document-type categories) demonstrates that the full MADP configuration with Human-in-the-Loop supervision attains 98.5% document-level accuracy. Additionally, we present a comprehensive sustainability analysis showing that our hybrid AI+HITL approach reduces CO2 emissions by 69%, energy consumption by 69%, and water usage by 63% compared to traditional manual processing. Benchmark comparisons of multiple LLM backends (Granite-Docling, Mistral-Small, DeepSeek-OCR) provide practical insights for deployment in production environments.

Summary

Main Finding

MADP (Multi-Agent Document Processing) is a modular pipeline that combines CNN-based classification, document parsing, LLM extraction, and a Human-in-the-Loop (HITL) Validator with a Prompt Fine Tuning with Feedback Inheritance (PFTFI) loop. In production (955 real invoices through Jan 2026) it achieved a 97.0% end-to-end automation rate, an ablation-validated document accuracy up to 98.5% with HITL, and projects ≈70% FTE reduction for a 100,000-invoice/year use case. The architecture also delivers substantial sustainability benefits versus fully manual processing (CO2 −69%, energy −69%, water −63%).

Key Points

Architecture: five agents — Classificator, Splitter, Parser, Extraction (LLMs), Validator — plus PFTFI feedback loop to incorporate human corrections into prompts/configuration without retraining models.
Production results: 955 documents processed; 926 (97.0%) handled fully by MADP; 29 (3.0%) routed to non-AI fallback for severely degraded/unrecognized formats.
Accuracy / Ablation (stratified 100-doc subset, 20 categories): Baseline (direct LLM on PDF) 60.0% → +Classificator 65.0% → +Splitter 72.5% → +Parser 90.0% → +Validator+PFTFI (automated) 92.5% → Full MADP + HITL 98.5%. The Parser provided the largest single improvement (+17.5 pp).
Classificator: ResNet-18 (first three blocks frozen, 4th fine-tuned on headers) — 95.3% accuracy on 5,000-doc test across 150 suppliers.
Parser: Docling-based, converts layouts into hierarchical markdown and reduces token count by ~35% vs raw OCR text — large positive effect on downstream LLM extraction.
Extraction: prompt-engineered LLM extraction with parallel-extractor/consensus mode; outputs JSON with per-field confidence scores.
Validator & PFTFI: atomic consistency checks (arithmetic, date, VAT/currency), thresholded routing to humans (typical thresholds 80–90%). Human corrections are captured and used to update prompts/configs; updates are versioned and applied to pending similar documents.
Human vs manual speed: average human review via validation GUI ≈45 seconds per flagged document; fully manual processing ≈120 seconds.
LLM backend trade-offs (selected results): Mistral-Small-3.2 — F1 92.9%, Precision 89.8%, Recall 98.2%, 17.8s/doc (chosen for production); DeepSeek-OCR — Precision 96.8% (best), 6.02s/doc; Granite-Docling — F1 77.5%, 7.76s/doc; fastest variant (DeepSeek-OCR-Regolo) 3.63s/doc with lower F1 (77.8%).
Sustainability: hybrid AI+HITL pipeline reduces CO2 emissions by 69%, energy consumption by 69%, water use by 63% relative to purely manual baseline (claimed in paper).
Operational resilience: modular design allows graceful degradation (route to manual/legacy workflows), and selective human validation controls risk of hallucination.

Data & Methods

Datasets:
- Production deployment: 955 real-world documents (invoices/statements) processed through Jan 2026; >50 suppliers; languages: Italian, English, German, French (plus two Arabic supplier samples).
- Ablation subset: stratified 100 documents (5 per each of 20 supplier/document-type categories).
- Classifier test: 5,000 documents across 150 supplier categories for ResNet-18 evaluation.
Models & components:
- Classificator: ResNet-18 (ImageNet pretrain; partial fine-tuning on header crops).
- Parser: Docling library for layout, table detection; outputs hierarchical markdown reducing tokens ~35%.
- Extractor: prompt-engineered LLMs; supports parallel backends + consensus voting.
- Validator: rule-based checks (format, arithmetic, cross-field consistency) + HITL GUI.
- PFTFI: generates structured feedback from human corrections, updates prompts and parser configs, versioning, applies updates to pending similar documents without retraining.
Metrics:
- Document-level accuracy: fraction of documents with all required fields correct.
- Field-level F1/precision/recall.
- Automation rate (fraction handled end-to-end by pipeline).
- Human intervention rate and human review time.
- Processing time per document (inference latency).
- Sustainability metrics: CO2, energy, water consumption (methodology claimed but details brief in paper).
Experiments:
- Production deployment logging (955 docs).
- Ablation study to quantify contribution of each component.
- LLM backend benchmark comparing F1/precision/recall and per-doc latency.
- Operational projection for 100k invoices/year to estimate FTE reductions and sustainability impacts.

Implications for AI Economics

Labor substitution and cost savings:
- Reported ≈70% FTE reduction for a 100k-invoice/year workload — direct labor cost savings are large and material for enterprises with document-intensive operations. Human reviewers are focused on edge cases, reducing per-document labor cost.
- Faster human review (45s vs 120s) further lowers marginal human labor costs on flagged documents.
Capital & operating trade-offs:
- Savings must be balanced against integration, engineering, and recurring inference costs (LLM inference compute, parser/classifier hosting). PFTFI avoids repeated model retraining (reducing retraining compute and data labeling costs), shifting investment toward prompt engineering, orchestration, and stable validator logic.
- LLM selection drives economics: higher-accuracy models (e.g., Mistral-Small) can reduce human review rate (lower variable labor), but incur higher per-doc latency and potentially higher inference cost. Faster/cheaper models (DeepSeek variants) lower throughput cost but increase human review burden due to lower F1.
- Parallel extraction/consensus increases compute per document but may reduce costly human interventions; tradeoff should be tuned by marginal human labor cost vs compute cost.
Risk management & compliance economics:
- HITL + deterministic validator checks reduce operational risk and exposure to hallucinations — economically important in regulated or audit-heavy settings (fewer costly errors, penalties, or rework).
- Versioned prompt/config updates increase auditability of corrections vs opaque model retraining — can lower compliance overheads and legal risk.
Environmental externalities and procurement decisions:
- The reported CO2/energy/water reductions are large and relevant to firms with ESG targets. Organizations can factor these savings into total cost of ownership (TCO) and procurement choices for AI systems.
- However, sustainability gains depend on system configuration (model choices, on-prem vs cloud, data-center energy mix). LLM inference energy costs remain significant; PFTFI's avoidance of retraining materially reduces the biggest energy sinks (retraining).
Scalability & marginal economics:
- High automation rates (97%) imply low marginal human cost per additional document; the system scales economically as volume increases.
- For lower-volume customers, fixed integration and engineering costs may dominate; payback depends on throughput and labor rates.
Operational levers to optimize economics:
- Tune HITL thresholds by field criticality and supplier reliability: higher thresholds reduce risk but increase human review; find equilibrium to minimize total cost (compute + human).
- Choose LLM backends per workload window (mix high-accuracy models for critical suppliers, faster models during peaks).
- Use PFTFI to amortize improvements across pending queues — accelerates benefit capture without retraining costs.
Limitations and risk to economic estimates:
- Production dataset is modest (955 documents) — projected 100k/year FTE reduction is modeled from observed behavior and may vary by document diversity, languages, and image quality in other organizations.
- Sustainability claims rely on the paper's accounting; practitioners should recompute using their cloud/on-prem energy mixes and model-selection choices.
- Upfront engineering/integration costs, vendor lock-in, and maintenance of prompts/configs are real ongoing expenses that affect TCO.
Practical recommendation for decision-makers:
- Run a pilot with the organization’s document mix, measure human intervention rate and per-doc inference costs for candidate LLMs, and use PFTFI or equivalent to avoid costly retraining cycles.
- Include lifecycle energy and labor savings in ROI and ESG evaluations.
- Balance model accuracy vs latency/cost depending on whether human labor or compute is the dominant marginal cost in your organization.

If you want, I can (a) extract the numeric tables (ablation and LLM benchmarks) into a CSV or spreadsheet-friendly format for economic modeling, or (b) produce a short ROI template to estimate payback for your specific invoice volumes, labor rates, and chosen LLM costs.

Assessment

Paper Typedescriptive Evidence Strengthmedium — The paper reports concrete production metrics (97.0% full-pipeline automation on 955 documents) and a stratified ablation (100 docs) that support system performance; however, the FTE reduction and sustainability claims rely on an operational scenario and modelling rather than randomized or externally validated causal identification, and evaluation samples are modest in size and scope. Methods Rigormedium — The authors use a sensible engineering evaluation: production deployment, stratified ablation, and multi-backend benchmarks plus human-in-the-loop validation; nevertheless, sample sizes for controlled evaluation are small, selection and measurement biases are not fully ruled out, and details on statistical uncertainty, error analysis across document types, and external replication are limited. SampleProduction deployment dataset: 955 real-world documents processed through January 2026; ablation/evaluation subset: stratified 100-document sample (5 documents for each of 20 supplier/document-type categories); and an operational scenario extrapolation for 100,000 invoices/year used to estimate FTE and sustainability impacts; benchmarking performed across multiple LLM/OCR backends (Granite-Docling, Mistral-Small, DeepSeek-OCR). Themesproductivity human_ai_collab IdentificationPerformance and operational comparisons: production deployment metrics on 955 real-world documents, an ablation study on a stratified 100-document subset (5 docs × 20 supplier/document-type categories) comparing full MADP vs component-removed configurations, and an operational scenario estimate (100,000 invoices/year) contrasted with a manual-processing baseline; sustainability gains are computed by modelling energy/CO2/water use relative to manual processing. No randomized assignment or quasi-experimental counterfactual is reported. GeneralizabilityEvaluation focuses on invoice/document processing in a specific enterprise context and may not generalize to other document types (contracts, forms, legal documents)., Production sample (955 docs) and ablation sample (100 docs) are modest and may not capture long-tail formats, languages, or rare edge cases., Results depend on chosen LLM/OCR backends, prompt fine-tuning method (PFTFI), and the implemented Human-in-the-Loop workflow; different toolchains or lower-quality OCR could change outcomes., FTE and sustainability estimates are modelled from an operational scenario rather than measured across multiple organizations, limiting external validity., Potential vendor-, region-, or supplier-specific formatting biases (e.g., language, PDF quality, invoice standards) may restrict transferability.

Claims (9)

Claim	Direction	Confidence	Outcome	Details
Operational analysis on a production use-case scenario of 100,000 invoices per year indicates a potential reduction of Full-Time Equivalent (FTE) requirements by approximately 70%. Employment	positive	high	Full-Time Equivalent (FTE) requirements	n=100000 approximately 70% reduction 0.18
Production deployment on 955 real-world documents processed through January 2026 achieves a 97.0% full-pipeline automation rate. Adoption Rate	positive	high	full-pipeline automation rate	n=955 97.0% full-pipeline automation rate 0.18
Only 3% of documents required non-AI fallback in the production deployment. Adoption Rate	positive	high	proportion requiring non-AI fallback	n=955 3% requiring non-AI fallback 0.18
Ablation evaluation on a stratified 100-document subset demonstrates that the full MADP configuration with Human-in-the-Loop supervision attains 98.5% document-level accuracy. Output Quality	positive	high	document-level accuracy	n=100 98.5% document-level accuracy 0.18
MADP combines deep learning-based classification and parsing with large language model extraction while maintaining accuracy through selective human validation. Output Quality	positive	high	maintenance of accuracy via selective human validation	0.18
The system integrates five specialized agents—Classificator, Splitter, Parser, Extraction, and Validator—together with a Human-in-the-Loop mechanism and a Prompt Fine Tuning with Feedback Inheritance (PFTFI) approach. Other	positive	high	system architecture components (agent integration)	0.3
Prompt Fine Tuning with Feedback Inheritance (PFTFI) is a novel approach introduced in this work. Other	positive	high	novelty of PFTFI method	0.03
A comprehensive sustainability analysis shows that the hybrid AI+HITL approach reduces CO2 emissions by 69%, energy consumption by 69%, and water usage by 63% compared to traditional manual processing. Other	positive	high	CO2 emissions, energy consumption, water usage	CO2 emissions reduced by 69%; energy consumption reduced by 69%; water usage reduced by 63% 0.18
Benchmark comparisons of multiple LLM backends (Granite-Docling, Mistral-Small, DeepSeek-OCR) were performed to provide practical insights for production deployment. Adoption Rate	positive	high	comparative performance of LLM backends for deployment	0.09

A hybrid multi-agent system automates almost all invoice processing: deployed MADP handled 97% of real-world documents end‑to‑end and achieved 98.5% accuracy in a stratified test, enabling an estimated ~70% reduction in labor and ~69% lower CO2 and energy footprints versus manual workflows.