0 cumulative citations
View corpus contextA deterministic tax-calculation engine worsened after-tax outcomes in simulated multi-agent financial advice, while retrieval-augmented context made no measurable difference; this suggests explicit tooling can introduce conflicting optimization signals that degrade performance in practice.
Retrieval-augmented generation vs. deterministic tax computation in multi-agent financial advisory: A 2x2 factorial experiment
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
In simulated multi-agent wealth-advisory scenarios, enabling a deterministic tax optimization engine unexpectedly reduced average tax savings relative to disabling it, while adding retrieval-augmented knowledge had no significant effect.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Tax-loss harvesting demonstrates consistent benefits to long-term portfolio growth; yet implementing it efficiently often involves complex considerations that are specific to the holdings within that portfolio and the individual who owns it. We introduce a custom capital gains calculation engine and a RAG-retrieved vector store of market advisory reports to provide context for a multi-agent trade recommendation system. We investigate the effects of each context provider on the quality of recommendations, measured by relative capital gains incurred during portfolio liquidation. A 2x2 repeated-measures ANOVA revealed a significant main effect of the tax optimization engine ($F(1,29) = 9.17$, $p = .005$, $η^2_p = .240$): enabling the engine reduced tax savings by approximately 55 percentage points relative to the no-engine conditions. The RAG main effect was not significant ($p = .841$), nor was the interaction ($p = .553$). The RAG-only condition achieved the highest descriptive mean tax savings (47.7%), and the baseline condition performed second-best (30.6%), suggesting that the pre-trained language model's internalized financial knowledge may be sufficient for competent tax-loss harvesting recommendations without explicit tooling. These results indicate that augmenting LLM agents with domain-specific computation engines does not guarantee improved performance and may introduce conflicting optimization signals.
Summary
Main Finding
Augmenting a multi-agent, LLM-based financial advisor with a deterministic capital-gains/tax optimization engine reduced after‑tax performance: enabling the tax engine produced significantly lower tax savings (mean −16.03% when on vs. 39.14% when off; F(1,29)=9.17, p=.005, η2p=.240). Retrieval‑Augmented Generation (RAG) from a vectorized advisory/report store had no significant main effect (p=.841) and did not interact significantly with the tax engine (p=.553). Descriptively, the RAG‑only condition achieved the highest mean tax savings (47.7%) and the no‑tool baseline was second (30.6%), suggesting the pretrained model’s internalized finance knowledge can be sufficient for competent tax‑loss harvesting in this setup, and that adding a deterministic tax tool can introduce conflicting signals that worsen outcomes.
Key Points
- Experimental design: within‑subjects 2×2 factorial toggling (Tax Engine: on/off) × (RAG: on/off) across the same 30 synthetic portfolios.
- Conditions and descriptive mean tax‑savings (%) (n=30 each):
- Baseline (no tools): 30.55% (SD 99.16)
- RAG only: 47.73% (SD 45.98)
- Tax engine only: −11.03% (SD 114.96)
- Tax engine + RAG: −21.02% (SD 161.94)
- Primary statistical results:
- Tax engine main effect: F(1,29)=9.17, p=.005, η2p=.240 (medium–large effect).
- RAG main effect: F(1,29)=0.04, p=.841, η2p=.001 (ns).
- Interaction: F(1,29)=0.36, p=.553, η2p=.012 (ns).
- Pairwise (Tax Tools off vs on): t(29)=3.03, pbonf=.005, Hedges’ g=0.67.
- Robustness checks:
- Excluding outlier portfolios (IQR bounds) left N=23; tax‑engine effect persisted (F(1,22)=4.38, p=.048, η2p=.166).
- Dollar‑value tax savings ANOVA aligned with percentage results (tax engine F=5.47, p=.026).
- Authors report additional nonparametric checks (Friedman) given skewness/variance but primary conclusions were stable.
- Architecture: LangChain-based multi-agent orchestrator with specialized worker agents (Holdings Analyst, Research, Tax, Mathematics, Guidance); toggles controlled agent access to deterministic tax computation and to a vector DB of market/tax advisory reports for RAG.
Data & Methods
- Design: 2×2 within‑subjects factorial experiment, 30 synthetic taxable brokerage account scenarios (each evaluated under all 4 conditions → 120 observations).
- Portfolios: 5–20 positions, values $25k–$500k, each with at least one unrealized loss to allow harvesting; holdings and history deterministically seeded.
- System under test: a multi‑agent LLM orchestration pipeline. Agents could use:
- A custom capital‑gains calculation / tax‑optimization engine (deterministic tooling with lot tracking and wash‑sale constraints).
- A vector store RAG of market/tax advisory reports to provide contextual documents to agents.
- Primary outcome: Tax Savings (%) = (Tbaseline − Trecommended) / Tbaseline × 100%, where Tbaseline is projected tax liability under no‑trade baseline.
- Statistical analysis: repeated‑measures (within‑subjects) 2×2 ANOVA for tax savings percentage; Bonferroni‑corrected pairwise comparisons; sensitivity analyses (outlier exclusion, dollar‑value outcome, nonparametric checks).
Implications for AI Economics
- Tooling does not guarantee improvement: adding deterministic, domain‑specific computation (tax engine) can harm outcomes when integrated with learned LLM agents—likely because of misaligned objectives, conflicting optimization signals, or coordination failures in multi-agent workflows.
- LLMs' internalized priors matter: pretrained LLM policies may already encode adequate heuristics for some domain tasks (here, tax‑loss harvesting), so external tools should be integrated only when they improve alignment and do not contradict latent model reasoning.
- Integration design is crucial: deterministic engines should be coupled with clear, consistent objective functions and failure‑mode handling (e.g., arbitration layers, calibrated constraint‑enforcement, unified loss functions or reward shaping) to avoid the tool producing incompatible recommendations.
- Evaluation & deployment guidance:
- Always run controlled A/B tests and within‑portfolio (paired) evaluations before deploying tool augmentations to clients.
- Monitor distributional effects and variance—the conditions with the tax engine showed much higher variance and extreme adverse cases.
- Interpretability and audits: deterministic tools can increase explainability but also create brittle interactions; require traceable decision paths and unit tests for tool outputs.
- Policy and product implications:
- Regulators and firms should not assume deterministic compliance tools automatically improve outcomes when combined with LLMs—governance must include integration testing and performance metrics for hybrid systems.
- For retail/advisory products, human oversight and staged rollout are advised, especially for tax recommendations that have legal/regulatory consequences (wash‑sale, reporting).
- Research directions:
- Diagnose failure modes: when and why did the tax engine recommendations increase tax liabilities? (e.g., incorrect lot selection, conflicting cost functions, timing/trade sequencing issues).
- Explore coordination architectures: arbitration/meta‑agent that reconciles deterministic outputs and LLM proposals; constrained generation techniques; reward/penalty terms to align agents with tax objectives.
- Test on real historical portfolios and incorporate transaction costs, slippage, and full wash‑sale mechanics; examine long‑horizon wealth effects, not only single‑event tax savings.
- Investigate methods to calibrate LLM priors to deterministic engines (e.g., fine‑tuning, supervised alignment, tool‑use training with joint objectives).
- External validity caveats: synthetic portfolios, simulated environment, relatively small N (30) with high outcome variance. Findings should be replicated on larger and real client datasets before broad generalization.
Overall takeaway: augmenting LLM‑based financial advisors with domain tools (deterministic tax engines) can backfire unless careful integration, objective alignment, and robust testing are performed; RAG alone did not harm or help significantly and in this experiment the pretrained model often matched or outperformed the hybridized setups.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The design has strong internal control (within-subjects, deterministic seeding, power calculation, appropriate ANOVA and robustness checks), producing a clear, statistically significant main effect for the tax engine; however, evidence is limited to simulated portfolios (N=30), high outcome variability, potential implementation/tooling confounds, and absence of real-world validation, reducing external credibility.
Methods Rigormedium — The paper reports a pre-specified within-subjects factorial design, power calculation, appropriate repeated-measures ANOVA, Bonferroni correction, outlier sensitivity analysis, and a dollar-value robustness check; weaknesses include reliance on synthetic scenarios, incomplete reporting of agent/LLM training and inference settings, limited description of retrieval corpus and tax-rule fidelity (wash-sale, transaction costs, jurisdictional details), and potential implementation bugs or optimization-signal conflicts that are not instrumented or decomposed.
Sample30 synthetic taxable brokerage account scenarios (within-subjects), each holding 5–20 securities, total account values $25,000–$500,000, each with ≥1 unrealized capital loss; deterministic seeding ensured identical holdings across four experimental conditions, yielding 120 condition-level observations (30 portfolios × 4 conditions); multi-agent recommendations produced by a LangChain-based orchestration with specialized sub-agents and toggled access to a custom capital-gains calculation engine and a vectorized retrieval store of advisory/tax documents.
Themeshuman_ai_collab innovation
IdentificationWithin-subjects 2×2 factorial experiment on 30 simulated client portfolios with deterministic seeding to hold portfolio holdings constant across four conditions (baseline, Tax Engine only, RAG only, Tax Engine + RAG); causal inference is based on within-portfolio comparisons using repeated-measures ANOVA and Bonferroni-corrected pairwise tests.
GeneralizabilitySynthetic portfolios may not reflect the distribution and complexity of real retail client accounts (position concentration, illiquidity, non-public assets)., Results reflect a single implementation (specific LLM backbone, agent prompts, tax engine logic, and retrieval corpus) and may not generalize to other models or toolchains., Potential simplifications/omissions in modeling transaction costs, market impact, execution constraints, and full jurisdictional tax rules (wash-sale nuances) limit real-world applicability., Small sample (30 portfolio scenarios) and high outcome variance reduce confidence in population-level effect sizes., Simulated, single-period evaluations (unclear multi-period horizon) may not capture dynamic effects of repeated harvesting over time.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Enabling the deterministic tax optimization engine significantly reduced tax savings compared with conditions in which the engine was disabled. Consumer Welfare | negative | Percentage reduction in projected federal capital-gains tax liability relative to a no-trade baseline |
Reading fidelity
high
Study strength
medium
|
n=30
approximately 55 percentage points
|
| The tax optimization engine's detrimental effect corresponds to a medium-to-large within-subjects effect. Consumer Welfare | negative | Percentage tax savings |
Reading fidelity
high
Study strength
medium
|
n=30
Hedges' g = 0.67
|
| Retrieval-augmented generation did not significantly affect tax savings. Consumer Welfare | null_result | Percentage tax savings |
Reading fidelity
high
Study strength
medium
|
n=30
η²p = .001
|
| There was no significant interaction between the tax optimization engine and RAG, indicating that the engine's negative effect did not depend on whether RAG was enabled. Consumer Welfare | null_result | Percentage tax savings |
Reading fidelity
high
Study strength
medium
|
n=30
η²p = .012
|
| The RAG-only condition had the highest descriptive mean tax savings, while the no-component baseline had the second-highest mean. Consumer Welfare | positive | Mean percentage tax savings |
Reading fidelity
high
Study strength
low
|
n=30
47.73% for RAG-only versus 30.55% for baseline
|
| When the tax optimization engine was enabled, mean tax savings became negative, indicating that the engine's recommendations increased tax liability on average. Consumer Welfare | negative | Mean percentage tax savings relative to the no-trade baseline |
Reading fidelity
high
Study strength
medium
|
n=30
−11.03% and −21.02% mean tax savings
|
| The finding that the tax engine reduced tax savings remained significant after excluding portfolios with extreme tax-savings values, whereas RAG and the interaction remained non-significant. Consumer Welfare | negative | Percentage tax savings |
Reading fidelity
high
Study strength
medium
|
n=23
η²p = .166 for the tax-engine effect
|
| The analysis of tax savings in dollar terms converged with the percentage-based results: the tax optimization engine significantly reduced tax savings, while RAG and the interaction were not significant. Consumer Welfare | negative | Tax savings amount in dollars |
Reading fidelity
high
Study strength
medium
|
n=30
η²p = .159
|