Model-switch timing is governed by learning speed: when model improvement follows a power law the optimal evaluation time scales as T^{1/(1+α)}, so learning-curve shape — not immediate cost — primarily determines when to stop experimenting. A sequential testing rule that anticipates local trends achieves near-oracle performance in theory and in a real-world credit-scoring deployment, avoiding premature discard or mistimed one-shot tests.

Main Finding

When new features become available only for a small, growing subset of observations, the optimal timing to stop experimenting and evaluate a challenger model is driven primarily by the shape of its learning curve. Under a power-law learning curve with exponent α and a finite horizon T, the oracle evaluation time scales as T^{1/(1+α)}. Costs (data collection, training/validation, deployment/switching) determine whether switching is profitable. Importantly, full knowledge of the learning curve is not required: any procedure that stops on the appropriate statistical scale (roughly T^{2/3} in the model-free guarantee) and makes reliable switch/discard decisions attains near-oracle regret O(T^{2/3} √log T). A practical sequential (look-ahead) evaluation algorithm that uses local learning-curve trends attains near-oracle performance in theory and in a real-world credit-scoring study, and is more stable than greedy or one-shot evaluation rules.

Key Points

• Problem setup
  • Incumbent model f_I trained on large historical data with feature set I.
  • New features C\I become available only for newly collected samples; challenger f_C must be trained on a small and growing full-feature dataset.
  • Decision epochs: at selected times the challenger is retrained and the decision-maker chooses one of three actions: switch (deploy f_C), discard (stop and keep f_I), or continue (collect more full-feature data).
• Economic performance gap
  • Define G(t) = E[gain(f_C trained at last epoch)] − E[gain(f_I)], where the gain is a bounded monetary-return function ℓ(·,·). G(t) drives the switching decision.
• Costs
  • Include per-sample data-collection/validation costs, training/compute/engineering costs (may have fixed and variable parts), and fixed deployment/switching costs.
  • Costs shape whether experimentally-observed improvements justify switching even if the challenger outperforms statistically.
• Learning-curve structure
  • Assume power-law improvement with exponent α (learning speed).
  • Oracle analysis: optimal training/evaluation time scales like T^{1/(1+α)} — faster learning (larger α) implies earlier stopping.
• Algorithmic results
  • Model-aware oracle benchmark characterizes economically optimal timing and profitability.
  • Model-free guarantee: any algorithm that stops on the T^{2/3} scale and reliably decides switch/discard gets O(T^{2/3} √log T) regret vs full-foresight.
  • Proposed practical algorithms:
    • One-shot rule: fixed-time evaluation (works only if timing matches true scale).
    • Greedy sequential: evaluate repeatedly and act on current estimates (prone to premature discards from noisy early signals).
    • Look-ahead sequential: use local learning-curve trends to anticipate future gains; empirically and theoretically most stable and near-oracle.
• Empirical validation
  • Real-world credit-scoring case study: look-ahead rule approaches oracle value in both early- and late-switching regimes and is more stable than greedy or one-shot rules.

Data & Methods

• Theoretical framework
  • Finite-horizon, discrete-time model with T time steps and constant batch flow: at each time step t, n new full-feature samples arrive (cumulative N_t = t·n).
  • Decisions occur at a subset of steps (decision epochs); challenger is retrained only at epochs.
  • Value measured by expected monetary gain function ℓ(·,·); define the expected economic performance gap G(t).
  • Cost structure includes per-sample and fixed components; deployment cost is incurred once upon switching.
• Learning-curve assumption and analysis
  • Power-law learning curve: performance (or error) scales as a power of sample size; exponent α captures learning speed.
  • Oracle optimization yields evaluation time ∝ T^{1/(1+α)}; switching profitability depends on costs relative to accumulated prospective gains.
• Regret and algorithm design
  • Regret measured relative to a full-foresight oracle that knows the entire G(t) path.
  • Show that stopping at T^{2/3} scale plus reliable tests gives O(T^{2/3} √log T) regret without knowing α.
  • Develop a look-ahead sequential evaluation algorithm: at each epoch, estimate local learning-curve trend to forecast near-term improvement and decide switch/discard/continue accordingly.
• Empirical methods
  • Applied to a credit-scoring dataset where alternative features accrue over time.
  • Compared one-shot, greedy sequential, and look-ahead sequential rules against the oracle benchmark; assessed monetary gains and stability (sensitivity to noisy early estimates).

Implications for AI Economics

• Timing decisions hinge on learning dynamics, not just instantaneous performance: firms should estimate or approximate learning speed α to schedule evaluations efficiently.
• Cost accounting is essential: even if a challenger eventually outperforms, per-sample collection and retraining costs plus deployment costs can make switching unprofitable if improvement is slow.
• Practical governance rules
  • Avoid purely greedy policies that react to noisy early estimates (risk of discarding promising challengers).
  • Avoid rigid one-shot evaluations unless the evaluation time matches the problem’s natural scale.
  • Prefer sequential, look-ahead evaluation cadence that uses local trends to anticipate gains—this balances exploration costs and deployment delays and is robust when α is unknown.
• Operational and investment guidance
  • When purchasing alternative data or investing in pipelines, evaluate expected learning speed (α) as a first-order determinant of the required evaluation horizon.
  • Use the T^{1/(1+α)} scaling as a planning heuristic for how long to collect full-feature samples before committing resources to a final decision.
  • For firms that cannot estimate α reliably, rules that target the T^{2/3}-type scale (model-free guarantee) provide provable near-optimality in a broad class of problems.
• Broader policy and valuation
  • Valuations of alternative data and ML upgrades should internalize data acquisition, validation, retraining, and switching costs, not just asymptotic predictive gains.
  • Regulatory and compliance costs (e.g., validation/audit requirements) can materially shift the switching-profitability threshold; including them in the cost model is necessary for realistic governance.
• Managerial takeaway
  • Principled model governance should jointly consider evaluation cadence, decision confidence, and optimism about future learning against the cost structure. Slow but eventual improvement does not automatically justify ongoing experimentation.