Large language models beat rule-based journal-entry tests at spotting tax-related ledger anomalies and offer readable explanations to guide auditors, though real-world deployment hinges on dataset representativeness, false-positive tradeoffs and operational integration.

Main Finding

Prompt-engineered large language models (LLMs), when fed heterogeneous journal-entry data plus dataset-level context and an Isolation Forest (IF) anomaly hint, can detect accounting anomalies at least as well as classic unsupervised baselines and far better than rule-based Journal Entry Tests (JETs). LLMs also produce natural-language explanations, improving interpretability for auditors. Prompt design (global statistics + IF hint) is critical: statistics drive recall, IF hints improve precision.

Key Points

• Problem: Traditional JETs produce many false positives and miss subtle irregularities; supervised labels are scarce in real ledgers. The task is unsupervised anomaly (fraud) detection on multi-type journal-entry (JE) records.
• Datasets:
  • Synthetic benchmark (Gronewald et al. 2024): controlled evaluation (sample of 5,000 posting IDs, 1% anomaly rate ≈ 50 anomalies).
  • Real-world anonymized general ledger from an industry partner: lacks posting IDs (transaction-level treated), no ground-truth labels — evaluation uses pseudo-labels from client-verified JETs.
• Models compared:
  • Rule-based JETs (baseline)
  • Isolation Forest (unsupervised ML baseline)
  • Multiple open and proprietary LLMs (Gemma-2B, Gemma-7B, Llama-3.1-8B, Mistral-8B, GPT-5-mini, etc.)
• Representative quantitative results (Synthetic dataset):
  • JET: precision 0.53, recall 0.90, FP = 942
  • Isolation Forest: precision 0.61, recall 0.98, FP = 169
  • Mistral-8B (best LLM): precision 0.90, recall 0.98, F1 = 0.94, FP = 12
  • Gemma-7B: recall 0.99, FP = 68 (very high recall)
  • Other LLMs show varied trade-offs (some high recall but many FPs)
• Real-world anonymized results (AuditCopilot prompt):
  • Isolation Forest: precision 0.30, recall 0.96
  • Gemma-7B (AuditCopilot): precision 0.89, recall 0.78, FP = 32 (high precision)
  • Gemma-2B (AuditCopilot): recall 0.98, precision 0.30 (high recall)
  • Prompt ablations: removing IF increases FP dramatically; removing global statistics severely collapses recall.
• Method specifics:
  • Prompt tuning only; no model weight updates. Inputs are compact JSON-like JE records.
  • System prompt augmented with dataset statistics (means, percentiles), user/account frequencies, and IF score/hint.
  • LLM outputs strict JSON: {"anomaly": 0|1, "explanation": "..."} — explanations intended to be actionable.
• Explainability: LLMs provide natural-language rationales which auditors find useful, though explanations may not perfectly reflect model decision boundaries.
• Practical considerations: Preference for open-weight LLMs (privacy), but risks include hallucinations, prompt sensitivity, pseudo-label bias, and governance/compliance challenges.

Data & Methods

• Framing: Unsupervised anomaly detection on multi-type JE data (numerical, categorical, text).
• Input representation: Each JE or transaction provided as a JSON-like record containing date, posting_id (when available), currency, user, tax rates, amounts, account codes, memo text. Prompts include dataset-level context (transaction counts, amount percentiles, user/account counts).
• Core procedure:
  • Use prompting (system + instance) to convert each transaction into a binary anomaly decision plus free-text explanation.
  • Augment prompt with Isolation Forest output (anomaly hint + score) and global statistics to improve discrimination and grounding.
  • No fine-tuning of LLM weights; behavior controlled via prompt engineering ("prompt tuning").
• Baselines and evaluation:
  • Baselines: standard JETs (rule-based) and Isolation Forest (unsupervised ML).
  • Metrics: precision, recall, F1, true/false positives/negatives, reported per dataset and prompt variant.
  • For the anonymized ledger, labels are pseudo-labeled via JET rules for evaluation; synthetic dataset has controlled injected anomalies.
• Ablation studies:
  • Removing IF hint → large FP increase.
  • Removing global statistics & IF → major collapse in recall.
• Privacy & model selection: experiments emphasize open-weight models (LLaMA family, Gemma, Mistral) to avoid sending confidential data to closed APIs.

Implications for AI Economics

• Labor and productivity:
  • Potential to reduce auditors' review burden by cutting false positives (e.g., Mistral-8B FP=12 vs JET FP=942 in synthetic test), raising effective productivity per auditor.
  • Hybrid workflows (human + LLM) may shift auditors’ tasks toward higher-level investigation and oversight, changing demand composition for audit labor (less repetitive screening, more judgment-heavy work).
• Cost-benefit trade-offs:
  • Model choice matters: some LLMs favor recall (catch more anomalies) at FP cost; others favor precision. Firms must value missed-anomaly costs vs. time spent on false positives to choose models/thresholds.
  • Using open-weight LLMs may lower data-sharing costs and privacy compliance burdens, affecting vendor market shares and procurement decisions.
• Market effects and productization:
  • Opportunity for new AI-augmented audit products (AuditCopilot-style tools) that bundle LLM inference, classical anomaly scores, and governance controls.
  • Demand for standardized, high-quality labeled audit datasets will grow; supply constraints (privacy, heterogeneity) will create premium for curated, anonymized benchmarks.
• Governance, regulation, and liability:
  • Regulators and auditors will likely require transparency, reproducibility, and audit trails for model outputs — increasing demand for explainability and model-stability testing.
  • Liability rules and certification regimes may emerge for AI-assisted auditing tools, affecting adoption speed and pricing.
• Externalities and risks:
  • Overreliance on LLM outputs can create automation bias; false negatives in high-stakes settings have systemic risk (misstated financials, tax fraud).
  • Prompt sensitivity and model updates imply maintenance costs (continuous monitoring, revalidation) which must be budgeted.
• Research & policy priorities (economic research agenda):
  • Quantify net cost savings from reduced FP burden vs. model inference and governance costs in real audit workflows.
  • Study labor reallocation: how many low-skill audit hours are displaced vs. upskilled roles created?
  • Market structure analysis: consequences of dominant LLM providers vs. open models for audit software competition and pricing.
  • Design regulatory frameworks balancing innovation and audit reliability (certification, mandatory benchmarks, disclosure rules).

Suggested immediate next steps for practitioners and economists: - Run controlled field trials to estimate real-world time/cost savings per flagged item and calibrate thresholds to local audit-cost structures. - Invest in benchmark datasets spanning industries/ERPs to measure external validity and craft regulation-compliant evaluation standards. - Incorporate governance costs (retraining, monitoring, human-in-the-loop reviews) into ROI models before deployment.

If you want, I can (a) extract a concise table of the key numeric results from the paper, (b) draft a short economic model to estimate firm-level savings from FP reduction, or (c) propose an experimental design to quantify labor impacts in audit teams.