A transformer trained on patent classification codes can forecast novel technological pairings years — sometimes decades — ahead by detecting collective shifts in how technologies are described, and it also improves representations for common patent analysis tasks.

Main Finding

The authors show that impending technological combinations leave an early, detectable trace in the collective language of patents. By embedding IPC technology codes directly into a transformer model’s vocabulary (TechToken) and measuring “context similarity” (CS) between code embeddings, they can predict first-time co-occurrences of technologies (i.e., novel recombinations) years—often decades—before those combinations appear in patents. The signal is corpus-level and emergent (not attributable to individual documents or inventors). TechToken also yields state-of-the-art patent embeddings that improve performance on standard patent tasks.

Key Points

• Innovation as recombination: forecasting novel combinations of existing technologies (IPC codes) reduces the unprecedented to a structured link-prediction problem (adjacent possible).
• Linguistic convergence precedes technological convergence: the textual contexts of two technologies begin to resemble each other well before they co-occur in a patent.
• TechToken method:
  • Expands a language model’s vocabulary with tokenized IPC codes (technological tokens).
  • Fine-tunes the LM so attention operates between patent words and IPC tokens, producing per-patent IPC-code embeddings that capture polysemy/context-specific meanings.
• Context Similarity (CS):
  • CS between two IPC codes is measured as cosine similarity among their embeddings.
  • For TechToken, CS is computed as the average similarity of the top 1% most similar embedding pairs (captures closest senses while reducing noise).
  • CS is computed over 1-year windows (plots often use a 3-year sliding average).
• Empirical performance:
  • TechToken substantially outperforms baseline embedding strategies and other LMs (including much larger LLaMA 3.1 8B) at predicting first co-occurrences.
  • Example AUC-ROC (class-imbalance 0.005%): TechToken BERT (IPC) ≈ 0.936; LLaMA Patents ≈ 0.856; BERT4Patents ≈ 0.725; fine-tuned BERT4Patents ≈ 0.765.
  • Predictive signals can be observed on average up to ~20 years before first co-occurrence.
• Robustness checks:
  • Use of a Chung–Lu random graph null model to compute expected co-occurrence counts and z-scores; high-z pairs are treated as realized innovations to focus on statistically significant novel combinations.
  • Multiple training/test splits and a stricter out-of-sample test (using 2019–2023 to predict first co-occurrences in 2024) to avoid leakage.
• Broader utility: TechToken embeddings improve downstream patent tasks (IPC classification, citation prediction, title–abstract matching), suggesting better semantic grounding from including structured labels in attention.

Data & Methods

• Data:
  • Patent corpus with IPC (International Patent Classification) labels and textual fields (abstracts, titles); temporal windows spanning at least 2006–2024 for various experiments.
  • Focus on code pairs that had never co-occurred up to the training window, then checking first co-occurrence in future windows.
• Embedding strategies:
  • Average Embedding: compute patent text embeddings with a standard LM, then average embeddings of patents associated with an IPC code → single static embedding per IPC.
  • TechToken: add IPC codes as tokens in LM vocabulary; during fine-tuning, IPC tokens appear alongside patent text, producing per-patent IPC token embeddings (captures context-dependent meaning).
• Measuring CS:
  • For two IPC codes, collect all per-patent token embeddings from the relevant time window, compute pairwise cosine similarities, then average the top 1% highest similarities to define CS for that year.
• Forecasting evaluation:
  • Construct candidate IPC pairs that never co-occurred before the training window.
  • Use CS from training window as classifier score to predict which pairs will become “realized innovations” in a later test window.
  • Define “realized innovation” by ranking code-pair z-scores (observed minus expected co-occurrences under Chung–Lu null, scaled by sqrt variance) and labeling top-K pairs as positive (controls class imbalance).
  • Report AUC-ROC across time-shifted 5-year test windows and at different class-imbalance thresholds (e.g., 0.005%).
• Baselines:
  • BERT4Patents (base), BERT4Patents further fine-tuned, LLaMA 3.1 8B fine-tuned on patents (LLaMA Patents) using alternative embedding methods.
• Additional tasks:
  • Evaluate TechToken embeddings on IPC classification, citation prediction, and title–abstract matching vs. prior state-of-the-art.

Implications for AI Economics

• Forecasting the adjacent possible: provides a practical, data-driven method to detect which technological recombinations are becoming feasible before they materialize—valuable for policymakers, industrial strategy, and R&D planning.
• Early signals for policy and investment:
  • Governments and firms could use CS-based monitoring to identify emerging cross-domain synergies and target early-stage support or investment.
  • Helps prioritize infrastructure, standards, or regulation in domains where linguistic convergence signals rapid convergence risk/opportunity.
• R&D portfolio and firm strategy:
  • Firms can detect nascent complementarities and guide acquisitions, alliances, hiring, or internal R&D allocation toward technologies whose language contexts are converging.
• Mapping technology trajectories and diffusion:
  • CS trajectories provide quantitative measures of conceptual convergence and can be incorporated into models of technology diffusion, capability accumulation, and sectoral evolution.
• Complement to network-based and bibliometric methods:
  • TechToken’s language-grounded signal captures emergent semantic change not visible from co-occurrence networks alone; combining signals (text + network metrics + citations) should improve forecast robustness.
• Methodological transfer:
  • The approach (embedding symbolic classification codes as tokens) generalizes beyond patents—e.g., scientific taxonomies, medical codes, industry/product classifications—allowing forecasting in other knowledge domains.
• Cautions and limitations:
  • Patent coverage bias: not all innovations are patented; sectoral differences in patenting behavior can bias signals.
  • IPC granularity: IPC codes are coarse; important micro-innovations inside codes may be missed; conversely, IPC assignment practices may introduce artifacts.
  • Language-driven confounders: increases in CS may reflect discourse trends, regulatory events, or increased attention rather than underlying technological feasibility—causality is not guaranteed.
  • Model and data vintage: embeddings depend on the training/fine-tuning data and cutoff dates; real-time deployment requires continual updates.
  • Statistical thresholds and choice of top-k/top-% affect sensitivity/specificity; operational use requires calibration to policy or investment risk appetite.
• Research opportunities:
  • Integrate CS signals into economic models of innovation, endogenous growth, and industrial policy evaluation.
  • Combine with firm-level, inventor-network, and market data to link early linguistic convergence to commercialization outcomes.
  • Explore multi-source fusion (patents + publications + grants + news) to triangulate signals and reduce false positives.
  • Use TechToken-style tokenization for domain-specific ontologies to forecast recombinations in science, medicine, and digital platforms.

Summary takeaway: embedding structured technology labels into transformer attention (TechToken) reveals a robust, corpus-level linguistic precursor to technological recombination that can meaningfully improve forecasting of novel innovations and has actionable implications for economic policy, corporate strategy, and further research in the economics of innovation.