Making predictive models first-class tools for LLM agents speeds up work: a pilot system that calls a small pricing model inside an LLM workflow generated priced proposals in under 10 minutes versus multiple hours. The pricing tool—trained on 70 real and human-verified synthetic examples—shows strong in-sample predictive performance, but narrow data and a single pilot limit claims about wider productivity gains.

Main Finding

Exposing pre-trained statistical ML models as callable tools within LLM agent workflows (Machine Learning as a Tool, MLAT) yields practical gains: it enables contextual, agent-controlled invocation of quantitative models, improves interpretability and composability of ML predictions, and—empirically via the PitchCraft pilot—reduces end-to-end proposal generation from 3+ hours to under 10 minutes while producing a pricing model that generalizes well (test R2 = 0.807, MAE = $3,688).

Key Points

• MLAT pattern: register a pre-trained ML model as a stateless, low-latency tool in an LLM agent’s tool registry; the agent extracts a schema-constrained feature vector and invokes the model when appropriate, then reasons about the returned prediction.
• Formalized workflow: feature extraction via structured output parsing (Gemini JSON schemas) → ML tool call (fθ) → agent contextual reasoning → structured output generation.
• PitchCraft case study: two Gemini agents (Research Agent and Draft Agent). Research Agent produces structured features with provenance; Draft Agent extracts features, calls an XGBoost pricing endpoint, reasons about the prediction, and outputs a JSON-constrained proposal mapped to a Google Docs template.
• Model performance (N = 70): XGBoost trained on 40 real + 30 LLM-generated, human-verified synthetic records. Results: training R2 = 0.937, CV R2 = 0.816 ± 0.060, test R2 = 0.807, test MAE = $3,688 (≈22.6% relative MAE vs. mean price $16,309). Inference latency <100ms.
• Small-data techniques: human-in-the-loop synthetic data generation conditioned on empirical distributions, group-aware splitting to avoid leakage (multi-phase clients grouped), aggressive feature reduction (final x ∈ R8; tech_stack one-hot encoded), ablation showing integration_complexity is highly influential.
• System design principles: schema-bridged inputs, stateless and fast endpoints, prediction transparency (confidence/metadata), model-agnostic registration, and agent-controlled invocation.

Data & Methods

• Dataset
  • N = 70 records: 40 real agency deals (22 industries) + 30 LLM-generated synthetic records that were human-verified and edited as needed.
  • Target: project price (right-skewed). Feature set: client_revenue, est_duration_weeks, pain_severity_score (1–5), integration_complexity (1–5), phase (1–4), tech_stack (no_code/low_code/custom → one-hot resulting in 8 input dimensions).
• Synthetic data procedure
  • LLM conditioned on real data summary statistics (means, SDs, correlations, ranges) to generate additional records.
  • Human review to remove implausible entries; synthetic records labeled with unique client groups.
  • Post-checks to confirm preserved correlation structure.
• Leakage controls
  • Group-aware splitting (GroupKFold/GroupShuffleSplit) to ensure all phases of the same client fall wholly in train or test; final split 56 train / 14 test.
• Modeling
  • Model: XGBoost regression chosen for small-data tabular robustness and fast inference.
  • Conservative hyperparameters: n_estimators=50, max_depth=3, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, reg_alpha=0.1, reg_lambda=1.0, min_child_weight=3.
  • CV: three-fold GroupKFold respecting client groups.
  • Comparisons: Ridge regression baseline (CV R2 ≈ 0.565) underperformed XGBoost.
  • Ablation: dropping integration_complexity reduced CV R2 markedly (0.816 → 0.599), indicating feature importance.
• Structured-output integration
  • Gemini JSON schemas used as the contract between agents: research schema supplies ML feature fields + provenance/confidence; draft schema consumes ML output and generates structured proposal including pricing_section.total_price and value_justification.
  • ML model exposed as a FastAPI REST endpoint; one-hot encoding applied at inference; endpoint latency <100ms.

Implications for AI Economics

• Productivity and time-to-value
  • Example impact: reducing proposal generation from multiple hours to under 10 minutes materially lowers labor costs per proposal and enables higher lead-response speed—an empirically important driver of conversion. Firms can scale outreach and personalization with the same or fewer staff.
• Labor substitution vs. augmentation
  • MLAT represents a complementarity: LLMs handle contextual reasoning and narrative, ML models handle quantitative estimation. This division can shift worker tasks from manual estimation and document assembly toward oversight, relationship work, and exception handling—raising per-worker productivity while preserving human-in-the-loop control.
• Value capture and pricing strategy
  • Embedding predictive pricing within an agentic workflow enables rapid, consistent, and explainable price quotes. Faster, consistent quoting can increase conversion rates and improve revenue capture; however, model errors (bias or systematic under/overpricing) risk lost margin or reputation, so governance matters.
• Lowering barriers for domain ML
  • MLAT reduces integration friction: models can be swapped/upgraded without changing agent logic, lowering switching costs and enabling continuous improvement. The approach also shows small-data ML (augmented by carefully verified synthetic records) can be economically viable for niche business tasks.
• Market structure and competition
  • Widespread adoption of MLAT-like architectures could compress time-to-proposal across service industries, intensifying competition on speed and personalization. Firms that combine domain data, validated small-data models, and agent workflows may obtain sustainable advantages.
• Costs, risks, and governance
  • Deployment costs are modest per-call (stateless REST endpoints, sub-100ms latency), but non-trivial costs arise from data curation, synthetic-data verification, and model validation. Risks include overfitting from synthetic records, distributional drift, and erroneous pricing decisions—requiring monitoring, human oversight, and conservative model outputs (prediction intervals/confidence).
• Research and policy opportunities
  • Need for empirical work on long-run effects: conversion lift from ML-informed quotes, wage/role shifts in affected occupations, and market-level price dynamics. Policy considerations include transparency requirements where automated pricing materially affects consumer outcomes and auditability of combined LLM+ML decisions.

Caveats and limitations - Small N (70) and reliance on LLM-generated synthetic data—mitigated here by human verification and group-aware CV—but external validation on larger, out-of-distribution datasets is needed. - Results come from a single pilot (Legacy AI’s PitchCraft); generalization to other domains and scales should be tested. - Economic impacts depend on adoption, regulation, and downstream behavioral responses (client acceptance of machine-informed pricing).

Overall, MLAT formalizes a pragmatic division of labor between quantitative ML models and reasoning LLMs, offering an actionable architecture that can increase productivity, enable modular upgrades, and lower operational friction for business tasks that combine numeric estimation with contextual narrative.