Professional-looking dashboards persuade LLM agents to act on unknowable futures: across 12 models adding an (even fabricated) market panel lifts commitment from ~6.5% to ~54%; a 540-case fine-tune of a 3B model can eliminate such false confidence but the safeguard breaks under subtle prompt or format changes.

Main Finding

An LLM agent’s decision to act (answer vs. decline) on inherently aleatoric, unknowable questions is driven far more by the authoritative appearance of accompanying evidence than by the evidence’ truth. Across 12 frontier models, adding a professional-looking indicator panel raises the rate of committing to a directional call from 6.5% (bare question) to 54.0%. Critically, panels whose numeric fields are entirely fabricated induce the same rise in commitment as genuine data. The failure sits at the action gate (whether to ANSWER), is separable and trainable (a 3B model fine-tuned on 540 synthetic cases learned to decline), but is fragile to output-format and prompt changes.

Key Points

• Effect size
  • Commitment (choosing ANSWER) rises from 6.5% (no panel) to 54.0% with richer panels across 12 models.
  • Fabricated partial-scramble panels (same entity, indicators from other dates) produce commitment statistically indistinguishable from real panels (rich 37.6% vs. scrambled 38.3% in the equity scrambled experiment).
  • Fully fabricated panels (nothing true in display except the question) still increase commitment relative to bare question (36.8% vs. 24.5% for a thin panel baseline).
• Localization of failure
  • The failure is at the act/don’t-act gate: stated probabilities barely change across evidence gradients and score worse than a climatological baseline (Brier analysis), so the model’s expressed beliefs do not explain the action change.
  • Matched answerable controls show models can answer answerable questions essentially perfectly but nevertheless commit on unknowable ones when presented with panel-like context.
  • Heterogeneity across models: some never commit, some always commit, some are strongly seduced; effect does not track capability.
• Trainability and fragility
  • Supervised fine-tuning a 3B model on 540 synthetic aleatoric cases (dice/coins/jars/timers) reduced commitment to 0% on the original 40 equity cases and transferred to three unseen domains in multiple runs.
  • The trained gate holds only when the response format affords explicit room to show reasoning; rigid/suppressed-output formats remove the gate, producing confident, wrong commitments (ablation run where the model committed on 48/48 unknowable items, assigning probabilities).
  • Robustness to differently-shaped prompts varies across training runs.
• Metrics & controls
  • Primary metric is commitment (ANSWER vs. CALL_TOOL/DECLINE). Discrimination measured via Youden’s J (decline|unknowable minus decline|answerable).
  • Degenerate strategies (e.g., “decline if future-tense” or “always decline/always answer”) are explicitly tested; a lexical confound (token “will” present only in unknowable items in transfer sets) shows the need for careful controls.
  • Case-clustered bootstrapping used for intervals; Brier and CORP decomposition used for calibration analyses.

Data & Methods

• Domains and oracle
  • Four domains: equities (primary), crypto (primary transfer), sports, and 10-day weather precipitation.
  • Aleatoric-unpredictable questions: short-horizon price direction, match winner, rain in ten days; equities & crypto outcomes sealed and verified near-chance ex ante; sports & weather constructed as unpredictable (weather carries caveats because ensemble forecasts have modest skill).
• Evidence gradient (matched conditions per case)
  • L0: bare question (no panel)
  • L1: current value + value ten periods prior
  • L2 (thin/rich variants): professional-looking panel with technical indicators (RSI, EMA, MACD, ATR, volume ratio, regime tag), or domain analogues
  • L2’: same full panel but belonging to a different entity (relevance control)
  • Additional scram/scramfull arms: indicators replaced with same-asset historical values or fully fabricated headers+indicators to isolate presentation from information.
• Protocol
  • Prompt: analyst scenario with tool web_search (explicitly limited to present/past), and three actions: ANSWER (probabilistic directional call), CALL_TOOL, DECLINE (explicitly say question is genuinely unpredictable).
  • Parsing: strict and semantic parsers applied consistently.
• Controls and baselines
  • Matched answerable questions attached to the same panels (e.g., is RSI > X?) to ensure models don’t simply decline everything.
  • Degenerate strategy baselines evaluated on same sets; showed pitfalls such as lexical shortcuts (e.g., “will”).
• Training intervention
  • Supervised fine-tuning of a 3B model on 540 synthetic aleatoric training cases (dice, coins, jars, timers), primarily synthetic domains unrelated to markets.
  • Multiple runs and ablations tested boundary conditions (response formats that either allow or suppress explicit stepwise reasoning).
• Calibration & scoring
  • Commitment as primary outcome (effect measured on actions, not just probabilities).
  • Brier score and CORP decomposition for probability calibration on sealed-outcome domains.
  • Youden’s J and case-clustered bootstrap for inference.

Implications for AI Economics

• Deployment risk in market-facing agents
  • Presentation/format can cause agents to take unjustified market actions even when the underlying content is fabricated. This creates attack surfaces for manipulation of automated advisory/trading agents via authoritative-looking dashboards or crafted retrieval results.
  • Because stated probabilities often do not reflect this shift in behavior, conventional calibration monitoring can miss dangerous shifts in decision-making.
• Design and evaluation recommendations
  • Evaluate agents on action-level metrics (commit/decline) under fabricated and adversarially authoritative contexts, not just probability calibration.
  • Include matched answerable controls and explicit defenses against lexical shortcuts (e.g., avoid systematic tense/token imbalances between classes).
  • Ensure response formats leave explicit room for stepwise reasoning or refusal; rigid templated outputs can suppress internal uncertainty and produce confident, wrong actions.
  • Train an explicit act/don’t-act gate using synthetic aleatoric cases as part of safety fine-tuning, but validate broadly: gate learning transfers but is context-fragile and can break under different prompting/output formats.
• Governance and regulatory relevance
  • Current assumptions baked into frameworks (more context => more reliable decisions) can be false: authoritative context can reduce safety. Regulators and auditors should require testing with fabricated-authority stimuli and require logging of when models decline vs. act.
  • Requirement for “abstain-when-unknowable” behaviors should be measured as an action-level property and stress-tested across presentation styles.
• Economic effects and research directions
  • Market impact: automated erroneous commitments could create spurious trading signals, amplify manipulation, and distort liquidity if deployed at scale; economic models of market quality should consider presentation-manipulation vectors.
  • Further research: robustness of gate across RLHF and preference-tuning regimes; interaction with multi-agent markets (how easily can adversaries exploit agents’ susceptibility to authoritative packaging); methods to certify abstention policies and to detect fabricated evidence displays.
• Practical mitigations for practitioners
  • Make abstention/decline explicit in the action interface and train models with varied output formats.
  • Log not only probabilities but the decision gate and chain-of-thought when allowed, and monitor divergence between internal reasoning and final action.
  • Test production agents with synthetic, fabricated displays (same-format, different-content) as part of pre-deployment QA.

Limitations worth noting - Primary sealed-outcome evidence is strongest for equities and crypto; sports and weather are weaker (weather especially: ensemble forecasts have modest skill and could make certain panels informative). - Transfer sets had a lexical confound (future-tense token “will” correlated with unknowable items); guarded against in some robustness tests but remains an open threat to generality. - Fine-tuned gate is context-fragile—robust, broadly generalizable abstention remains an open problem. - Study focuses on short-horizon aleatoric unpredictability; results do not directly generalize to epistemic unanswerability or to settings where additional tools (external observations/sensors) could legitimately resolve uncertainty.