A model-agnostic pipeline using LLM-driven semantic normalization and density-based clustering uncovers coherent research themes in ML and NLP conference abstracts. The method scales to large corpora and generates interpretable topic maps that can be used to track research frontiers and build indicators for economic analysis, though results require robustness checks across models and prompts.

Main Finding

Prompt-driven semantic normalization using large language models, combined with geometric (embedding + density-based clustering) analysis, provides a scalable, model-agnostic unsupervised framework that discovers coherent, human-interpretable research themes in large scientific corpora (demonstrated on ICML and ACL 2025 abstracts).

Key Points

• Framework treats an LLM as a fixed semantic inference operator guided by structured soft prompts to normalize abstracts into compact semantic representations that reduce stylistic variability while preserving conceptual content.
• Normalized representations are embedded into a continuous vector space; density-based clustering identifies latent themes without pre-specifying the number of topics.
• Cluster-level interpretation is performed via LLM-based semantic decoding to generate concise human-readable labels/descriptions for discovered themes.
• Experimental results on ICML and ACL 2025 abstracts produced coherent clusters that map to problem formulations, methodological contributions, and empirical contexts.
• The approach is scalable and model-agnostic: different LLMs/embedding models can be swapped into the pipeline without changing the overall method.

Data & Methods

• Input data: scientific abstracts (ICML and ACL 2025 in the experiments).
• Semantic normalization: apply a large language model with structured soft prompts to transform raw abstracts into normalized semantic representations (aim: reduce stylistic noise, retain core concepts).
• Embedding: convert normalized semantic representations into continuous vectors using an embedding model.
• Clustering: apply density-based clustering in the embedding space to detect dense regions corresponding to latent themes; the method does not require predefining the number of clusters.
• Interpretation: decode cluster content with an LLM to produce concise semantic descriptions / labels for each cluster.
• Evaluation: qualitative and cluster-coherence analyses showing clusters align with research problem types, methods, and empirical settings (as reported for ICML and ACL).

Implications for AI Economics

• Topic measurement and dynamics
  • Use the pipeline to generate high-resolution topic maps and time series for AI research areas (emergence, growth, decline).
  • Construct topic-level growth indicators (counts, share of publications, citation-weighted output) to measure the pace and direction of technological change.
• Mapping research frontier and competitiveness
  • Identify frontier topics and cross-field convergence (e.g., methods migrating from NLP to vision) to inform assessments of comparative advantage and specialization across institutions/countries.
  • Track which organizations/authors are driving new clusters to measure research leadership and returns to R&D investment.
• Linking research to economic outcomes
  • Map topics to downstream outcomes (patents, product introductions, industry adoption, labor demand) to study knowledge diffusion and productivity effects.
  • Use cluster assignments to define treatments in quasi-experimental designs (event-study or diff-in-diff) that estimate causal impacts of funding, regulation, or technology shocks on research direction and economic outcomes.
• Funding, policy, and strategic investment
  • Inform funding agencies about emerging methodological opportunities and gaps; allocate funding to under-explored but fast-growing themes.
  • Monitor effects of policy changes (e.g., data-access rules, export controls) on research composition and geographic distribution.
• Labor-market and human-capital analysis
  • Link topic maps to author career trajectories to study skill demand, retraining needs, and returns to specialization in specific AI subfields.

Caveats and recommended validation steps for AI economics use - Model and prompt sensitivity: results depend on the chosen LLM, soft-prompt design, and embedding model—perform robustness checks across models and prompts. - Cluster reliability: validate cluster stability (e.g., bootstrap, perturbations), and complement automatic labels with expert human validation for critical analyses. - Bias and representativeness: LLMs and corpora may reflect disciplinary, geographic, or language biases; adjust or stratify analyses accordingly. - Temporal modeling: the described pipeline is cross-sectional; extend to dynamic models (temporal embeddings, change-point detection) when analyzing trends or causal effects. - Interpretability for causal inference: use clusters primarily as descriptive or as a basis for constructing transparent variables; ensure careful identification strategies when using them in causal models.

Suggested practical steps for researchers - Build topic time series from cluster assignments and compute standard indicators (share of publications, citation-weighted output, concentration measures). - Link clusters to patents, grants, industry filings, or labor-market data to analyze diffusion and economic effects. - Run robustness checks with alternative embedding models and clustering algorithms (e.g., hierarchical, k-means) and validate with manual coding for a sample. - Consider using the method to pre-process and reduce dimensionality before causal estimation, then apply econometric methods (event studies, diff-in-diff, IV) for inference.