A lightweight LLM assistant that gives gist-level feedback improves decision accuracy and confidence under uncertainty and curbs unnecessary oversampling; verbatim feedback, by contrast, drives more extensive exploration, suggesting designers should match feedback granularity to the uncertainty profile of the task.

Main Finding

An LLM-based conversational assistant (SERA) that provides on-demand summaries during sequential information search improves decision accuracy and confidence—particularly in high-uncertainty environments. The format of feedback matters: gist (high-level) summaries encourage more efficient integration and reduced oversampling, while verbatim (detail-preserving) summaries encourage more extensive exploration. These effects imply that matching feedback granularity to the environment’s uncertainty structure can improve exploration–exploitation regulation.

Key Points

• Problem framed: many real-world choices require sequential information search and stopping under uncertainty; AI support typically targets final outcomes but may better serve users by scaffolding the search process.
• SERA: a conversational, LLM-driven assistant that (a) summarizes user-recorded notes on demand and (b) prompts brief monitoring (self-regulatory) reflections.
• Two feedback formats tested:
  • Gist: compressed, high-level meanings emphasizing what matters now.
  • Verbatim: literal, detailed recaps preserving attribute-level content.
• Environments (operationalized by how perceived information gain evolves during sampling):
  • Low-uncertainty (Decremental): predictable, steadily diminishing information gain.
  • Medium-uncertainty (Local Optimum): a local drop then rise in information gain (risk of stopping too early).
  • High-uncertainty (Random): no predictable pattern in information gain.
• Main behavioral patterns:
  • SERA (either format) → higher decision accuracy and greater confidence than no-feedback, with larger gains in higher-uncertainty settings.
  • Gist → more efficient integration, descriptive reduction in oversampling (i.e., fewer redundant samples; better stopping).
  • Verbatim → greater exploration (more sampling), which can help avoid local optima but may increase cognitive cost.
• Design implication emphasized by authors: feedback representation is a manipulable design lever; adaptive interfaces should match summary granularity to environmental predictability.

Data & Methods

• Overall design: two experiments (N1 = 54, N2 = 54) using a web-based search-and-stop decision task adapted from decisions-from-experience paradigms.
• Conditions: SERA-Gist, SERA-Verbatim, and a no-feedback baseline crossed with three uncertainty environments (decremental, local-optimum, random).
• Interaction flow: participants click to reveal sequential descriptive information for two options, record notes, request summaries from SERA, respond to a monitoring prompt, then continue sampling or stop and choose.
• SERA implementation:
  • Frontend: React.js; backend: Flask; data logged to Firebase.
  • LLM: OpenAI gpt-4o-mini; prompt framed the model as a neutral assistant with concise outputs. Parameters reported (temperature = 1, max tokens = 150, top_p = 0).
  • Summaries generated from participant-entered notes; format (gist vs verbatim) controlled via prompt framing.
• Outcomes measured:
  • Outcome effectiveness: decision accuracy, decision confidence.
  • Process regulation: number of samples, stopping behavior, integration patterns (efficiency vs oversampling), exploratory behavior.
  • Secondary: user perceptions and individual differences (to assess reliance patterns).
• Main results (reported descriptively in manuscript):
  • SERA improves accuracy and confidence relative to baseline; effect sizes increase with environmental uncertainty.
  • Gist reduces redundant sampling and supports efficient stopping; verbatim increases exploration.

Implications for AI Economics

• Design of decision-support tools in economic settings should go beyond final recommendations to actively scaffold the information search process; the representation of feedback materially affects exploration costs and welfare.
• Match feedback granularity to environment structure:
  • Predictable/diminishing-return environments (e.g., routine product comparisons, well-known financial instruments): favor gist summaries to reduce sampling costs, speed decisions, and reduce overload.
  • Environments with local optima or sparse but informative late signals (e.g., complex investment research, procurement with hidden features): favor verbatim or detail-rich summaries to encourage deeper exploration and avoid premature stopping.
  • Highly unpredictable markets: SERA-style prompts and summaries increase confidence and accuracy overall; a hybrid/adaptive policy that responds to observed information-gain patterns can balance exploration costs and decision quality.
• Platform and market applications:
  • E‑commerce search: gist summaries can lower search costs and abandonment; verbatim summaries can be used in expert-aimed workflows where exhaustive evaluation matters.
  • Financial advising and consumer finance: adaptive feedback could reduce under-search or overreliance and improve calibration of confidence in costly uncertainty.
  • Labor markets and procurement: conversational summaries can speed screening while verbatim support can be enabled when the environment indicates the risk of missing late-arriving important signals.
• Policy and welfare considerations:
  • Designers must consider cognitive costs and risk of overreliance; fluent conversational summaries may be persuasive and increase confidence—even when model outputs are uncertain—so transparency about system limits remains important.
  • Adaptive feedback that detects the payoff-structure and switches granularity can improve aggregate welfare but requires careful evaluation to avoid nudging users toward prematurely low-effort choices.
• Research & modeling opportunities for AI economics:
  • Formalize the value of information under different information-gain temporal structures and compute optimal feedback policies (when to provide gist vs verbatim).
  • Integrate user search-cost heterogeneity and attention limits into market-design models to predict impacts of different feedback strategies on social welfare, search duration, and market liquidity.
  • Evaluate long-run behavioral effects (habit formation, calibration of trust, exploitation of persuasive summaries) and the strategic consequences for platforms that choose different feedback policies.

Limitations to note for economic application: relatively small experimental Ns, lab-style descriptive information (not monetary gambles), single LLM and prompt family, and potential domain differences; results should be validated in larger, domain-specific field settings before widescale deployment.