Researchers use LLM research assistants more like collaborators than search engines, submitting longer, more complex queries and delegating tasks such as drafting and gap identification; experienced users become more targeted and engage citations more deeply, though simple keyword queries persist.

Main Finding

AI-powered research assistants (Asta’s PaperFinder and ScholarQA) are used not merely as search engines but as collaborative research partners: users submit longer, more complex, and task-oriented queries (including drafting, methodological guidance, and gap identification), treat generated reports as persistent artifacts they revisit, and — with experience — issue more targeted queries and engage more with cited evidence. The authors publish the Asta Interaction Dataset (AID) to support further study and evaluation.

Key Points

• Dataset released: AID contains 258,935 anonymized queries and 432,059 clickstream interactions (Feb–Aug 2025) from two deployed interfaces: PF (PaperFinder) and SQA (ScholarQA).
• Users formulate richer queries than traditional search:
  • Mean query lengths (words): PF ≈ 17.0, SQA ≈ 37.0, vs. Semantic Scholar (S2) ≈ 5.4.
  • More constraints, relations, and entities per query on Asta than S2.
• Query taxonomy introduced: 3 aspects (intent, phrasing style, criteria) with 16 intents (e.g., Broad Topic Exploration, Research Gap Analysis, Content Generation), 7 phrasing styles (e.g., Keyword, Natural Language, Complex Context), 6 criteria types (e.g., Methodology, Temporal), and 28 fields of study.
• Common behaviors and intents:
  • Keyword-style and broad exploration queries remain common, but many users ask for higher-level tasks (drafting methods, ideation, interpreting results).
  • Methodology-specific constraints are the most common search constraint on Asta (~42% PF, ~29% S2).
• Engagement and persistence:
  • Reports are persistent artifacts: report revisitation (SQA 50.5% of users, PF 42.1%) exceeds near-duplicate query submission (SQA 18.8%, PF 14.8%).
  • Non-linear reading patterns: many users expand non-consecutive sections; users skip introductions frequently.
• Learning effects:
  • As users move from first query → inexperienced (2–10) → experienced (>10), broad-topic queries decline and targeted/verification behaviors increase.
  • Evidence clicks (checking inline citations) rise with experience (e.g., +27% between 1st and 4th query); PF link-clicks drop among experienced users (-24%), implying more in-place consumption.
• Operational metrics:
  • Median session duration: PF ≈ 4 minutes, SQA ≈ 8 minutes.
  • Median model response time: PF ≈ 34 s, SQA ≈ 129 s.
• Methodological notes and limits:
  • Labels: 30k single-turn queries annotated using GPT-4.1 (structured decoding); taxonomy refined with Gemini-2.5-pro.
  • Statistical methods: two-sided t-tests (α=0.05), binomial logistic regressions with Benjamini–Hochberg FDR control.
  • Privacy: PII-filtered queries dropped; only hashed report IDs released; internal pseudonymous user IDs used for analysis but not released.
  • Possible selection/opt-in bias; taxonomy and automated labeling may introduce labeling noise; results drawn from one platform integrated with Semantic Scholar.

Data & Methods

• Data
  • Source: Asta (LLM-powered, retrieval-augmented system) integrated with Semantic Scholar.
  • Time window: February–August 2025.
  • Volume: 258,935 queries; 432,059 clickstream events.
  • Interfaces: PF (paper-ranked lists + short syntheses) and SQA (multi-section literature reports with inline citations).
  • Baseline comparison: Semantic Scholar (S2) query logs used for contrast.
• Preprocessing & Privacy
  • Bot filtering, session identification, PII removal; opted-in users; released dataset anonymized (no user pseudonyms).
• Labeling & Taxonomy
  • Iterative human+LLM workflow: manual inspection → LLM (Gemini-2.5-pro) suggestions → consolidation.
  • 30,000 single-turn queries labeled (intent, phrasing, criteria) using GPT-4.1 structured decoding.
• Analyses
  • Metrics: click-through rate (CTR: S2 link clicks), section expansions, evidence clicks, feedback (thumbs up/down) — CTR treated as primary success surrogate.
  • User-stage cohorts: single-query, inexperienced (2–10 queries), experienced (>10 queries) tracked longitudinally.
  • Statistical tests: two-sided t-tests (α=0.05); logistic regression predicting click behaviors with FDR control.
• Availability
  • Authors release AID (schema in appendix) to enable replication and further study.

Implications for AI Economics

• Productivity and task reallocation
  • Evidence that researchers delegate higher-level tasks (drafting sections, ideation, experimental design) suggests potential labor reallocation: routine search and synthesis may be automated, shifting researcher time toward creative, oversight, and validation tasks. This could raise per-researcher output but also change required skills (verification, prompt engineering).
• Skill-biased adoption and inequality
  • Experienced users gain more from the tool (better-targeted queries; more effective citation verification). If access or learning curves vary across institutions or researchers, tools may exacerbate productivity gaps across individuals and organizations.
• Platform competition, differentiation, and market structure
  • Integration of retrieval + LLM synthesis (and presentation choices like inline evidence & persistent reports) affects user behavior (in-place consumption vs. click-through). Product design, quality of grounding/citations, and dataset access may become key differentiation axes in the scholarly search/assistant market, increasing returns to platforms that successfully combine retrieval quality with trustworthy synthesis.
• Valuation & monetization signals
  • Persistent artifacts and repeated revisits suggest sustained value per query/report; firms could monetize via subscriptions, premium features (longer context windows, provenance tools), or enterprise analytics. Metrics like CTR, evidence-click rates, and revisitation can serve as monetization KPIs beyond simple query volume.
• Measurement of research output & impact
  • If AI assistants change how literature is discovered and synthesized, downstream citation patterns, knowledge diffusion speed, and even research agendas may shift. Economists should account for changes in search/synthesis friction when measuring productivity, collaboration networks, or the speed of cumulative innovation.
• Incentives, evaluation, and model design
  • The prevalence of methodology-specific and constraint-laden queries implies demand for models that support precise, verifiable outputs (structured citations, metadata filtering). Economic incentives favor models and interfaces optimized for verifiability and utility (not just fluency). Evaluation frameworks for AI in science should incorporate real-world interaction signals (e.g., click-through to sources, revisitation) and task success beyond static benchmarks.
• Data externalities and research diffusion
  • Released interaction logs (AID) create a public good enabling independent study of tool effects on research behavior and on market outcomes (adoption, inequality). At the same time, platforms collecting richer interaction data may gain strategic advantages (feedback loops improving models), creating data-driven network effects that could concentrate market power.
• Policy considerations
  • To ensure equitable access and to capture societal benefits, policymakers may consider supporting open-access tools/datasets or training programs for researchers less familiar with these assistants. Transparency requirements (provenance, citations) could reduce misinformation risks and support evaluation of scientific claims generated by AI.
• Research agenda enabled by AID
  • The dataset allows empirical study of economic questions: causal impact of AI assistants on individual researcher productivity, differential returns by field or career stage, firm-level adoption dynamics, and effects on scholarly communication and citation economies.

Limitations to bear in economic interpretation: dataset reflects opt-in users of one platform and one integration (Semantic Scholar), labels partly automated (LLMs), and the time window is limited — causal inference on productivity or welfare will require careful identification strategies and complementary data (e.g., publication outcomes, career-level indicators).