A new 142,808-conversation corpus shows that commercial LLM platforms produce systematically different user interactions and outputs when their native affordances (citations, thinking traces, code) are preserved; these differences affect intent satisfaction, citation strategies, and latency dynamics and are invisible to homogenized, single-platform benchmarks.

Main Finding

SHARECHAT is a large, multi-platform corpus of authentic chatbot conversations that preserves native platform affordances (citations, thinking traces, code artifacts) across five major commercial chatbots (ChatGPT, Perplexity, Grok, Gemini, Claude). With 142,808 conversations (660,293 turns) in 95 languages collected from public share-URLs (Apr 2023–Oct 2025), SHARECHAT reveals platform-specific user behavior, grounding strategies, and multi-turn dynamics that are obscured by single-interface or text‑only benchmarks.

Key Points

• Scale and scope
  • 142,808 conversations, 660,293 interaction turns, average 4.62 turns per conversation (longer than most prior corpora).
  • 95 languages (English 61.8%, Japanese 18.0%, others <3% each).
  • Mean assistant response length ≈ 1,115 tokens (Llama‑2 tokenizer).
• Platforms covered
  • ChatGPT (102,740 convs), Perplexity (17,305), Grok (14,415), Gemini (7,402), Claude (946).
  • Preserves platform-specific metadata when available (citations, timestamps, model version, thinking/code/analysis blocks).
• Collection & privacy
  • Discovery via Internet Archive (Wayback Machine) for share-URL patterns; rendering and scraping with Selenium; interactive elements (e.g., Claude “thinking” blocks) triggered programmatically.
  • PII removal via Microsoft Presidio with an automated LLM-based audit. IRB-approved.
• Quality & content signals
  • Lower measured toxicity than comparable datasets (overall LLM-turn toxicity: Detoxify 1.6%, OpenAI moderation 3.2%; user-turn rates also lower than WildChat).
  • Topics: information-seeking dominates (~39.6%), followed by “Other/Unknown” and technical help; per-platform specialization (Perplexity strongly search-oriented; Claude higher technical help; Grok more multimedia/social).
• Illustrative analyses (case studies)
  • Conversation completeness (Qwen3-8B pipeline to extract intentions and judge fulfillment): Claude highest full-completion rate (≈88%), ChatGPT ≈83%, Gemini ≈77%, Grok ≈74%, Perplexity ≈67%. LLM-based extraction achieved ~87% accuracy vs human labels.
  • Source grounding: Perplexity cites a broad set including Wikipedia, Reddit, NIH; Grok heavily cites X (Twitter) and news—indicating different retrieval paradigms (encyclopedic vs social/real-time).
  • Temporal/latency analysis: preserved timestamps reveal divergent response latency and engagement patterns across platforms (details in paper).
• Availability
  • Dataset publicly released (https://huggingface.co/datasets/tucnguyen/ShareChat).

Data & Methods

• Discovery and scraping
  • Query Wayback Machine for platform-specific share-URL patterns; crawl rendered pages using Selenium; parse into structured JSON.
  • Interact programmatically to reveal hidden elements (e.g., click thinking bars).
• Metadata & normalization
  • Extract turn sequences, timestamps (when present), model versions, view/share counts, citation URLs, thinking/code/analysis blocks.
  • Language detection with lingua-py; tokenization with Llama‑2 tokenizer.
• Privacy & filtering
  • PII redaction using Microsoft Presidio across multiple languages; redaction audit via automated LLM evaluator.
• Annotations & analyses
  • Toxicity: Detoxify and OpenAI Moderation API (language coverage caveats; different detectors show systematically different rates).
  • Topic classification: Llama‑3.1‑8B‑Instruct few-shot classifier (4‑bit quantized inference); validated vs human labels (82% accuracy).
  • Conversation completeness: three-stage pipeline (intention extraction, resolution classification, conversation-level scoring) using Qwen3‑8B; human validation on sampled conversations.
  • Source analysis: aggregate citation domains, compare distributions across platforms to reveal retrieval strategies.
• Limitations noted by authors
  • Self-selection bias: only publicly shared conversations (likely non-representative of all usage).
  • Platform coverage uneven (Claude sample small); some metadata absent on certain platforms (e.g., turn timestamps missing for some).
  • Anonymization may remove contextual signals; automated PII detection not perfect.

Implications for AI Economics

1. Platform design drives demand and user segmentation

   • Interface affordances (search citation, social media integration, “thinking” traces, code blocks) shape user tasks and loyalty. Economically, product differentiation creates niches (search-focused vs creative vs technical), affecting cross-platform substitution and pricing strategies.

2. Data as a durable competitive advantage

   • Platforms that preserve and collect richer, longer, multi-turn interactions (and affordance-specific metadata) produce higher-value training signals for RLHF and fine-tuning. This strengthens incumbents and raises entry barriers—data heterogeneity has direct implications for model quality and cost of replication.

3. Monetization and engagement value

   • Longer, denser assistant responses and multi-turn completions suggest higher engagement/utilitarian value per session. These signals can be monetized (subscriptions, premium features) and influence LTV (lifetime value) calculations and ad/productization choices.

4. Impact on measurement, investment, and regulation

   • Standard benchmarks that ignore platform affordances understate real-world performance differences. Investors and policymakers should account for interface effects when evaluating product claims, competition, or consumer harm.
   • Citation/source behaviors imply different misinformation/ liability profiles: social-media-grounded systems (Grok) may have higher recency but greater instability/bias risk; encyclopedic retrieval (Perplexity) favors authority but may lag on emerging events.

5. Labor and productivity economics

   • The dataset (multi-turn, task-resolving dialogs) enables quantification of AI-assisted productivity gains across domains (coding, writing, research). Differences across platforms indicate heterogeneity in productivity impacts and thus in willingness-to-pay by professionals vs casual users.

6. Market structure and dynamics

   • Specialized strengths (e.g., Claude for technical tasks, Perplexity for research) can produce differentiated demand curves and multi-homing behavior. Platform-specific improvements via continuous RL on proprietary logs can lead to widening gaps and potential winner-take-most outcomes.

7. Research & policy opportunities

   • SHARECHAT enables empirical estimation of: (a) elasticity of demand to interface features and response quality; (b) value of source-grounding to user trust and conversion/retention; (c) externalities from platform-specific misinformation propagation.
   • Policy relevance: data portability, transparency requirements about training/feedback data, and antitrust assessment should consider the role of proprietary interaction logs and affordance-specific data in entrenching market power.

Practical suggestions for economists and policymakers - Use SHARECHAT to estimate how specific affordances (citations, thinking traces, code blocks) affect willingness-to-pay and retention. - Model entry barriers by quantifying the marginal value of platform-specific logs for RLHF improvements. - Assess consumer harm and liability exposure by linking citation sources to factuality/instability metrics.

If you want, I can extract specific statistics (by-platform token counts, toxicity rates, or the completeness percentages in a table) or propose empirical designs using SHARECHAT for an economic study (e.g., estimating willingness-to-pay for citation-enabled answers).