Fine-tuned open-source language models sharpen antimalarial virtual screening: TxGemma-9B attains ROC-AUC ≈0.731 and LlaSMol-Mistral-7B achieves EF@1% ≈4.99, outperforming Random Forest/XGBoost and few-shot proprietary models; however, gains hinge on careful dataset curation and domain-specific fine-tuning rather than generic few-shot reasoning.

Main Finding

Fine-tuned, domain-specialised open-source LLMs (via parameter‑efficient fine‑tuning) substantially outperform classical cheminformatics baselines and frontier proprietary few‑shot models on out‑of‑distribution antimalarial virtual screening. Domain-specific fine‑tuning is effectively indispensable for reliable bioactivity prediction under structurally novel (Lo‑Hi dissimilarity) test conditions: TxGemma‑9B achieved ROC‑AUC 0.731 ± 0.005 and LlaSMol‑Mistral‑7B achieved the best prospective enrichment (EF@1% ≈ 4.99). In contrast, the same models in few‑shot/no‑fine‑tune settings collapsed to near‑random performance (e.g., TxGemma un‑fine‑tuned ROC‑AUC ≈ 0.499; Gemini 2.5 ≈ 0.53; OpenAI o3 ≈ 0.59).

Key Points

• Dataset and benchmark
  • Introduced Malaria‑Instruct: an instruction‑tuning dataset for antimalarial virtual screening built from ChEMBL Legacy Malaria with assay deduplication, temporal harmonisation, assay contextualisation, and chemically plausible negative sample augmentation.
  • Used rigorous Lo‑Hi dissimilarity splitting per assay (ECFP4 fingerprints; train–test max Tanimoto ≤ 0.40; train–validation ≤ 0.55) to force true out‑of‑distribution evaluation.
  • Primary operational metric emphasized: EF@1% (prospective hit enrichment); also reported ROC‑AUC, MCC, precision, accuracy.
• Models and results
  • Fine‑tuned open‑source models evaluated: Gemma‑2 (2B/9B), TxGemma (2B/9B), LlaSMol‑Mistral (7B).
  • Best performers: TxGemma‑9B (ROC‑AUC 0.731 ± 0.005), LlaSMol‑Mistral‑7B (EF@1% ≈ 4.99).
  • Classical baselines (Random Forest, XGBoost on Morgan fingerprints) and proprietary few‑shot models (Gemini 2.5, OpenAI o3) did not reach the fine‑tuned open‑source models' performance under the dissimilarity split.
  • Biomedical pretraining gave measurable gains at a given model scale; chemistry‑aware pretraining (LlaSMol) improved enrichment.
• Fine‑tuning vs few‑shot
  • QLoRA parameter‑efficient fine‑tuning with instructionized examples (70% few‑shot formatted during training) produced large gains; few‑shot prompting of large general‑purpose models was insufficient for reliable bioactivity discrimination in this setting.
• Experimental/ecological design
  • Conservative label curation: replicate concordance required for label inclusion; conflicting replicates discarded to prioritise label precision.
  • Negative augmentation: ECFP4 → PCA → t‑SNE selection of negatives neighboring active clusters to create realistic decision boundaries for confirmatory assays.
  • Closed‑source model evaluation limited to few‑shot API calls on subsamples (500 molecules) to emulate low‑resource token/monetary constraints.

Data & Methods

• Dataset: Malaria‑Instruct derived from ChEMBL Legacy Malaria; standardisation pipeline (charge neutralisation, parent extraction, tautomer canonicalisation); per‑assay contextual metadata for biological conditions and Plasmodium strains.
• Splitting: Lo‑Hi dissimilarity splitting per assay using ECFP4 (2048 bits, radius 2); thresholds: train–test Tanimoto ≤ 0.40; train–validation ≤ 0.55. Independent resplits for replicates.
• Negative augmentation: For confirmatory assays lacking inactives, select negatives from the PCA/t‑SNE neighbourhood of positives to restore positive:negative ratio while preserving chemical plausibility.
• Models and tuning:
  • Classical: Random Forest, XGBoost on 2048‑bit Morgan fingerprints (radius 2).
  • LLMs fine‑tuned with QLoRA: Gemma‑2 (2B/9B), TxGemma (2B/9B), LlaSMol‑Mistral (7B). Training mix: 70% few‑shot‑formatted examples (2–5 in‑context demos), 30% zero‑shot.
  • Few‑shot evaluation: 3/4/5‑shot conditions for un‑fine‑tuned open‑source and closed‑source models; closed‑source tested via API on subsamples.
• Evaluation metrics: primary operational metric EF@1% (hit enrichment); ROC‑AUC, MCC, precision, accuracy reported; validation sets held out as zero‑shot examples for final evaluation.
• Compute / reproducibility: Fine‑tuning performed using Colab Pro+; code, dataset, and fine‑tuning artifacts made available (GitHub, Zenodo links in paper). Reported experiments run with independent Lo‑Hi resplits to capture partition variance.

Implications for AI Economics

• Cost‑effectiveness of fine‑tuning small/medium open‑source models
  • Parameter‑efficient fine‑tuning (e.g., QLoRA) on 2B–9B models yields large performance gains at far lower compute/cost than reliance on largescale closed‑source models or brute‑force scaling. This reduces capital and operating costs for labs in resource‑constrained settings.
  • Open‑source domain‑specialised models can beat proprietary few‑shot LLMs for specialised tasks, implying better marginal value per compute dollar when paired with task‑specific fine‑tuning and curated datasets.
• Value of curated, domain‑specific datasets
  • Investments in careful curation (assay harmonisation, contextual metadata, realistic negative augmentation) produce outsized returns: they enable small/medium models to learn the fine‑grained structure–activity mappings necessary for out‑of‑distribution generalisation.
  • Funders and institutions should prioritise dataset creation and maintenance as high‑leverage investments relative to raw compute scaling.
• Trade‑offs: scale vs inductive bias vs data
  • Biomedical or chemistry‑aware pretraining provides inductive biases that can be more valuable than modest increases in parameter count. Economic decisions should weigh the marginal benefits of model scale against targeted pretraining and dataset augmentation.
• Access and equity considerations
  • Demonstrated viability of Colab Pro+ training and parameter‑efficient methods suggests democratisation pathways for drug discovery research in the Global South; lowering financial barriers can unlock locally relevant discovery pipelines (e.g., antimalarials).
  • However, there remain recurring costs (expert annotation, compute for fine‑tuning/inference, experimental validation) and legal/licensing considerations for models and datasets that funders must account for.
• Operational deployment economics
  • EF@1% improvements translate directly to experimental savings: higher enrichment reduces wet‑lab screening costs per discovered hit. Quantifying monetised value of EF gains would strengthen ROI arguments for dataset/fine‑tuning investments.
  • Closed‑source few‑shot evaluation constrained by API token costs and subsampling in the study—real deployments may incur considerable ongoing inference costs, whereas local fine‑tuned open‑source stacks allow predictable, lower marginal cost inference.
• Policy and strategic recommendations
  • Prioritise funding for public, high‑quality domain datasets and tooling for parameter‑efficient fine‑tuning to maximize societal benefit per dollar.
  • Encourage open‑source model development and permissive licensing to reduce dependency and monetary leakage to proprietary providers, especially for global‑health priorities.
  • Integrate economic assessments (compute + annotation + wet‑lab validation costs) into benchmarking to better inform procurement and funding decisions.

Limitations worth noting (for economic interpretation) - Exact compute hours, GPU types, and dollar costs are discussed in the paper (Table 4) but not reproduced here; precise ROI calculations will require those numbers plus local wet‑lab cost figures. - Study focuses on antimalarial assays and structurally novel generalisation; results may not transfer identically to other therapeutic areas without analogous datasets and curation. - Closed‑source models were only evaluated in few‑shot API mode (no fine‑tuning), so cost comparisons should account for that constraint.

If you want, I can: - extract the paper’s compute/time/cost figures (Table 4) and produce an ROI estimate (compute + annotation vs expected wet‑lab savings) for a hypothetical screening campaign, or - produce a short slide‑ready summary aimed at research funders highlighting recommended investments and expected payoffs.