A chatbot-based, image-augmented survey of student commuters shows cycling collapses in poor weather and transit use increases, while vision-equipped large language models can predict mode choice almost as accurately as conventional machine-learning benchmarks, reaching up to 71.5% five-class accuracy with tailored prompting.

Main Finding

A three-agent, auditable workflow (conversational data collection → structured processing → behavioral prediction) can collect image-augmented stated-preference travel data and enable multimodal LLM-based mode-choice prediction that matches or slightly exceeds conventional machine-learning benchmarks. Key empirical results: cycling falls sharply under adverse weather, transit use rises under snowy conditions; a random-forest benchmark reached 69.6% five-class accuracy, the best text-only zero-shot LLM reached 69.9% without task-specific training, and a vision-capable LLM configuration achieved the highest observed five-class accuracy at 71.5%. Prompt design (habitual-travel context, expert framing, few-shot examples) and visual inputs meaningfully affect LLM predictive performance.

Key Points

• Multi-agent workflow: three specialized agents (Data Collection, Data Processing, Data Modeling) coordinate an auditable pipeline from survey design to model evaluation.
• Data collection: chatbot-administered, image-augmented stated-preference survey with 92 student commuters across five weather scenarios → 454 respondent–scenario observations.
• Behavioral patterns: cycling highly sensitive to adverse weather; public transit usage increases notably under Snowy conditions.
• Modeling benchmarks:
  • Multinomial logit used for behaviorally interpretable estimates.
  • Random forest benchmark: 69.6% five-class accuracy.
  • Logistic regression and other ML baselines also evaluated.
• LLM experiments:
  • Nine locally deployed LLMs evaluated (model sizes ~2–35 billion parameters).
  • Evaluated across four zero-shot prompt-and-context conditions; extended experiments included persona conditioning, few-shot demonstrations, and vision-capable configurations using the same images shown to humans.
  • Best text-only zero-shot LLM: 69.9% five-class accuracy (no task-specific fitting).
  • Vision-enabled best configuration: 71.5% five-class accuracy (highest observed).
• Prompting insights:
  • Providing habitual travel information produced the most consistent improvements in LLM prediction accuracy.
  • “Expert” framing generally outperformed “Role-Play” framing.
  • Persona conditioning helped most when habitual-travel history was not available.
  • Few-shot prompting improved performance for several models; gains typically stabilized after a small number of examples.
• Limitations: small, non-representative sample (students), limited scenarios, results require validation on larger, diverse traveler populations.

Data & Methods

• Survey and sample:
  • Instrument: conversational chatbot delivering five predefined weather scenarios with photorealistic images; respondents reported mode choices per scenario.
  • Sample: 92 student commuters; 454 usable respondent–scenario observations.
• Multi-agent architecture:
  • Data Collection Agent: builds and deploys the chatbot survey, embeds images, aggregates raw responses and logs.
  • Data Processing Agent: de-identification, validation, recoding, feature construction, and quality flags; produces processed dataset for modeling.
  • Data Modeling Agent: fits discrete-choice models, trains ML classifiers, executes LLM-based prediction experiments, and evaluates results.
• Modeling approaches:
  • Econometric: Multinomial Logit (MNL) to estimate weather effects with behavioral interpretation.
  • Machine learning: Random forest, logistic regression, etc., used as predictive benchmarks (evaluation metric = five-class accuracy).
  • LLM-based prediction: nine locally hosted models (2–35B params). Configurations included:
    • Zero-shot with different prompt/context variants.
    • Persona conditioning (traveler personas).
    • Habitual travel history included vs. excluded.
    • Prompt framings (Expert vs. Role-Play).
    • Few-shot prompting with small numbers of demonstrations.
    • Vision-capable models ingesting the same weather images used in the survey.
• Evaluation:
  • Primary metric: five-class classification accuracy (modes).
  • Comparisons made between MNL, ML benchmarks, and multiple LLM configurations.
  • Analyses of how context inputs and prompting strategies influenced predictive performance.

Implications for AI Economics

• LLMs as predictive and synthetic agents:
  • LLMs can capture substantial predictive signal for discrete choice tasks without task-specific training; with careful prompting and context, they rival conventional ML models. This suggests potential for LLMs to serve as fast, low-data predictive tools in economic demand estimation and scenario analysis.
  • Persona and few-shot prompting can emulate heterogeneity and behavioral histories, supporting simulation of subpopulations or counterfactual agents in economic models.
• Multimodal context matters:
  • Inclusion of visual context (images representing environment/weather) improved accuracy in this study. For economic settings where decisions depend on visual/contextual cues (store crowding, product presentation, real-estate photos, weather, infrastructure), multimodal LLMs can add value over text-only inputs.
• Workflow reproducibility and auditability:
  • The multi-agent, modular approach provides an auditable pipeline (design → raw logs → processed dataset → models → evaluations). For economic research and policy work, such traceability is essential for replication, regulatory review, and model governance.
• Trade-offs with interpretability and validity:
  • Econometric models (e.g., MNL) retain interpretable behavioral parameters useful for welfare analysis and policy design; LLMs deliver competitive predictive accuracy but are less directly interpretable. Hybrid approaches (use LLMs for prediction, DCMs for policy inference; or LLMs to augment features for DCMs) may be productive.
  • LLM-based synthetic respondents are promising for scalable simulation, but empirical divergence from human behavior and distributional biases require careful validation before use in policy or economic inference.
• Practical considerations for deployment in economic modeling:
  • Prompt engineering and inclusion of past-behavior context materially influence outcomes — modelers must treat prompts and contextual inputs as part of model specification.
  • Small-sample validation (as here) is insufficient for generalizing to population-level forecasts; researchers and practitioners should validate on larger, representative samples and quantify uncertainty.
  • Governance: data privacy, reproducibility, and ethical implications (e.g., biases in LLM outputs) must be addressed before operational use in policymaking or market forecasting.
• Research opportunities:
  • Using multimodal LLMs to simulate demand under rare or hypothetical conditions (extreme weather, new product designs) where revealed data are sparse.
  • Integrating LLM-generated synthetic agents into economic agent-based models, while establishing calibration and validation protocols to align simulated behavior with empirical observations.
  • Systematic study of how different prompt framings, persona specifications, and demonstration sets alter economic predictions and policy-relevant counterfactuals.

Summary: This paper demonstrates a practical, auditable multi-agent pipeline that couples conversational, image-augmented data collection with structured processing and multimodal LLM prediction. Results show LLMs (especially when given behavioral history, appropriate framing, few-shot examples, or images) can approximate or exceed traditional ML benchmarks in mode-choice prediction, but small sample size and representativeness limit generalizability. For AI economics, the work highlights both opportunities (fast multimodal demand prediction, synthetic agents, reproducible pipelines) and cautions (need for validation, interpretability, governance).