Structured, ontology-driven retrieval does not raise raw QA accuracy for single-document financial questions but materially improves auditability: KDAF/CARP preserves provenance for every selected evidence item and increases citation-traceability F1 to 0.515 versus lexical baselines, making it valuable for regulated FP&A workflows despite higher construction cost.

Main Finding

Ontology‑driven, provenance‑aware graph retrieval (KDAF + CARP) does not improve raw question‑answering accuracy over strong lexical retrieval on single‑document filings questions, but it substantially improves auditability: KDAF yields higher citation traceability and enforces entity/company boundaries so that every retrieved evidence item is anchored to a complete provenance chain. The paper argues that this auditability — not marginal gains in accuracy — is the economic axis that can justify the cost of ontology construction in regulated finance settings.

Key Points

• Problem and motivation
  • FP&A and regulated finance require traceable, explainable, auditable outputs; fluency alone is insufficient for operational use.
  • Flat retrievers (sparse/dense) provide similarity scores that are not actionable as audit evidence.
• Proposed framework (KDAF)
  • Six iterative stages: (1) problem‑centric scoping (competency questions), (2) ontology bootstrapping → Minimum Viable Graph (MVG), (3) schema‑guided LLM extraction, (4) contextual knowledge representation (typed relevance + provenance), (5) hybrid human validation, (6) Context‑Aware Relevance Propagation (CARP) for retrieval.
  • Representations: OWL/RDF specification + property‑graph materialization (edge properties for traversal efficiency).
  • Contextual relation types (e.g., CausalDriver, SupportingEvidence, Correlational) carry propagation weights and confidence; provenance is modeled as traversable entity→filing→page→evidence chains.
• CARP algorithm (Stage 6)
  • Steps: seed identification, weighted propagation (ontology weights), contextual boundary detection (dynamic, depth‑relative thresholds), evidence selection.
  • Two‑stage selection: graph governs eligibility and provenance; final ordering is a composite where normalized lexical score dominates (weight 0.45 in the evaluated profile), with propagation score, concept coverage, and period coverage contributing remainder.
  • CARP emits a full retrieval trace (seed reasons, traversed edges, thresholds, provenance of each selected evidence).
• Evaluation (proof‑of‑concept)
  • Dataset: FinanceBench (145 FP&A questions).
  • Baselines: zero‑context LLM inference, BM25 sparse retrieval, concept‑weighted lexical retrieval (text + tables), and ungrounded graph traversal.
  • Main empirical findings:
    • Retrieval is necessary: zero‑context correctness 4.1% vs retrieval conditions 10–12%.
    • Accuracy parity: KDAF vs BM25 difference −0.007 (95% CI [−0.021, 0.000]) — no statistically significant accuracy improvement from structured retrieval on these questions.
    • Auditability gains: KDAF citation traceability F1 = 0.515, significantly higher than ungrounded graph traversal (+0.027, 95% CI [0.006, 0.050]) and BM25 (+0.052, 95% CI [0.024, 0.083]).
    • Company/entity boundary enforcement: graph‑structured retrieval (both ontology‑grounded and ungrounded) returned 0 of 426 and 0 of 424 evidence items from outside the question’s subject entity, versus 16.8% and 20.2% for lexical baselines.
    • Every selected graph‑based item resolves to a complete provenance chain.
• Limitations and caveats
  • Several KDAF stages (1,2,5) require domain experts and were not fully exercised in the public benchmark evaluation.
  • The evaluated CARP profile is hybrid and lexical‑dominated by design; relevance types and causal annotations were not instantiated in the benchmark (no narrative commentary in FinanceBench).
  • Evidence outside the selected budget is counted but individual non‑selection is not fully recorded (gap noted).
  • Results pertain to single‑document/filings style FP&A questions; multi‑hop causal tasks may show different tradeoffs.

Data & Methods

• Data
  • FinanceBench: 145 FP&A analytical questions (single‑document filings style).
  • Knowledge graph: schema instantiation (kdaf‑financebench‑provenance‑v1) with entity anchoring and provenance; contextual relevance types not present in this benchmark instantiation.
• Systems compared
  • Zero‑context LLM inference (no retrieval).
  • BM25 sparse retrieval baseline.
  • Concept‑weighted lexical retrieval over text and tables.
  • Ungrounded graph traversal (graph retrieval without ontology grounding/CARP).
  • KDAF instantiation with CARP (hybrid: graph eligibility + lexical‑heavy ranking).
• CARP implementation details
  • Propagation is multiplicative; per‑depth thresholds are relative to best score at that depth (adaptive frontier) with a floor.
  • Hard company boundary applied during traversal (prevents cross‑entity evidence).
  • Final ranking is an additive composite: lexical score (0.45) + propagation score + concept and period coverage.
  • Output: ordered evidence blocks plus a serialized retrieval trace recording seeds, traversals, thresholds, and provenance.
• Evaluation metrics
  • Answer correctness (binary or per benchmark scoring).
  • Citation traceability F1 (measure of auditability).
  • Statistical comparisons (differences with 95% confidence intervals reported).

Implications for AI Economics

• Cost vs benefit tradeoff
  • Building and maintaining ontology‑grounded KGs and the human validation processes entail substantial fixed and recurring costs (ontology engineering, expert time, hybrid validation tooling).
  • For single‑document filings QA where strong lexical retrievers match accuracy, the direct accuracy benefit does not justify that cost—unless auditability has monetizable value.
• Value of auditability in regulated finance
  • Auditability yields regulatory and compliance benefits: supports model risk management, enables independent review, and reduces regulatory/legal exposure. These benefits can translate into avoided penalties, lower oversight costs, and easier internal approvals—economic gains not captured by accuracy metrics alone.
  • The observed citation traceability improvement (F1 ≈ 0.52) quantifies a plausible reduction in assessor effort to validate system outputs, which can be monetized as time saved for controllers/auditors.
• Operational and labor impacts
  • KDAF’s enforcement of entity boundaries and guaranteed provenance can reduce the time spent on source chasing and manual data validation, potentially shifting FP&A labor from data collection toward higher‑value analysis.
  • However, onboarding (competency question elicitation, MVG bootstrapping) requires skilled staff; budgets must account for initial expert-hours and ongoing ontology evolution costs.
• Product and adoption considerations
  • Vendors and enterprise buyers should evaluate ROI not just by improvement in top‑line accuracy but by metrics that combine correctness + auditable provenance (e.g., cost per auditable correct answer, auditor time saved, compliance risk reduction).
  • In environments where regulatory traceability is mandatory, enterprises may be willing to pay the premium for ontology‑grounded systems even when accuracy is parity.
  • The hybrid CARP choice (lexical‑heavy ranking) highlights a product design lever: reweighting toward graph signals could improve content differences at potential cost of retrieval coverage; economic optimization depends on user priorities (accuracy vs auditable provenance).
• Research and product directions with economic relevance
  • Extend evaluations to multi‑hop causal FP&A tasks where structured reasoning may yield accuracy gains—this could change the cost/benefit calculus in favor of KGs.
  • Quantify auditability benefits directly: measure auditor time, decision‑maker confidence, and regulatory outcomes attributable to provenance‑rich retrieval.
  • Explore tooling and process innovations to reduce ontology‑engineering costs (semi‑automated MVG bootstrapping, improved human‑in‑loop interfaces), thereby lowering the threshold for positive ROI.
• Potential systemic effects
  • Widespread adoption of auditable LLM analytics could lower aggregate model risk across the financial sector, potentially reducing regulatory capital or oversight intensity over time—but it may also create demand for new roles (ontology stewards, provenance auditors) and vendor lock‑in for maintained ontologies and tooling.

Summary takeaway: In regulated finance, the economic case for investing in ontology‑grounded retrieval depends less on modest accuracy gains and more on measurable auditability benefits. Enterprises should evaluate structured retrieval investments using combined metrics that capture provenance, auditor effort saved, and regulatory risk mitigation, not accuracy alone.