Candle-based ML models fail to convert crypto predictions into profitable Binance Spot trading rules: tested strategies lost between ~2% and 44% after assumed costs, and a forensic audit revealed holdout contamination and missing raw results that undermine earlier claims.

Main Finding

Candle-based ML models that show strong discrimination on rare intraday extrema do not, in the tested Binance Spot settings, translate into positive executable policies once chronology, transaction costs, execution semantics, adaptive search, and artifact provenance are audited. Across four artifact-backed campaigns (mandatory-daily selector, local extrema minimum/maximum, Gurgul-inspired paired daily, and a consumed rotation control) none produced a deployable, positive-return trading rule — the operational decision for every tested protocol is NO_TRADE.

Key Points

• Primary negative results (all cost assumptions reported; 31 bps used as the main stress unless noted):
  • Mandatory daily selector (ten USDT pairs, 1-min candles): lost 6.72% over 19 July cycles at 31-bps completed-cycle cost (3 wins, 16 losses).
  • Local-minimum policy (BTC/ETH/SOL, 5-min candles): validation-selected local-minimum policy returned −1.79% in short July evaluation.
  • Local-maximum sell-to-cash/re-entry policy underperformed continuous holding by 2.80%; gross mean advantages were 11.11 and 12.21 bps — below stress thresholds.
  • Gurgul-inspired OHLCV-only daily adaptation (BTC/USDT & ETH/USDT): high ROC AUC (min/max: 0.874 / 0.896) but low average precision (0.134 / 0.116) and lost 44.30% over seven cycles versus −41.20% for cost-matched buy-and-hold.
• Predictive discrimination (ROC AUC) can coexist with economically negative policies when imbalance, low precision, timing, costs, and ordering effects are considered.
• Forensic audit downgraded a previously-cited One4All “30‑day holdout” diagnostic due to missing raw result directories, unpurged four-hour label horizon at splits, same-close entry semantics, and prior exposure of its dates — demonstrating how provenance issues can overstate claims.
• Extensive adaptive search and selection shaped outcomes: e.g., the mandatory-daily candidate was chosen after at least 946 enumerated candidates across several campaign summaries; One4All reported searches numbered 3,996 profiles. Such search budgets increase selection risk and reduce the evidential strength of apparent gains.
• All numerical results derive from stored deterministic simulators and archived models; no live trading or market interaction occurred. AI agents assisted only with literature retrieval, critique, artifact reconciliation, documentation, and packaging — not trading decisions.

Data & Methods

• Data sources and universes:
  • Binance Spot klines (OHLCV): various granularities and universes per campaign:
    • Local extrema: BTCUSDT, ETHUSDT, SOLUSDT, 5-min candles (Mar 2025–Jul 12, 2026).
    • Mandatory daily: ten USDT pairs (ADA, AVAX, BNB, BTC, DOGE, ETH, SHIB, SOL, TRX, XRP), 1-min candles (Mar 2025–Jul 19, 2026).
    • Paired daily (Gurgul-inspired): BTCUSDT & ETHUSDT daily candles (2021-06-01 to 2026-07-01 terminal open).
    • Rotation control: ten-pair daily candles (2021-06-01 to 2026-07-01).
• Targets and label construction:
  • Local extrema: centered extrema over 2b+1 windows (b ∈ {6,12,24}), plus a 40-bps rebound/pullback requirement.
  • Mandatory selector: realized four-hour policy outcome using +100 / −150 gross-bps target/stop.
  • Paired daily: separate next-open minimum/maximum labels over symmetric 7/14/21-day windows.
  • Rotation: monthly momentum-based signals with multiple lookbacks.
• Models and families:
  • Tabular and sequence models: logistic regression, SVM, Random Forest, HGB, Gaussian Bayes, MLP, CNN, GRU, LSTM, Transformer, hybrid architectures, and an ExtraTrees realized-net regressor (used for mandatory daily selection).
  • Cross-family selection, grid/threshold scans: many models/thresholds compared during validation (e.g., 14 models × multiple windows/quantiles → hundreds of validation policy comparisons).
• Evaluation and chronology safeguards:
  • Time-aware train/validation/evaluation splits, but several runs consumed prior periods for cross-family selection (documented). The paper uses a two-dimensional evidence taxonomy: artifact support (presence/absence of raw records) and data status (frozen prospective, model-specific later-period, consumed diagnostic, descriptive control).
  • Purging of label horizons was applied in several campaigns; where it was not (One4All), the diagnostic was invalidated.
• Execution semantics and cost model:
  • Simulation assumptions: entries use next-period open after decision on completed candle; stops/targets evaluated on OHLC bars; adverse stop-first tie rule when both reachable within a bar.
  • Cost stresses: 21 / 31 / 51 bps round-trip scenarios (31 bps primary). Paired-daily applies half cost multiplicatively per side; rotation charges half-cost per unit one-way turnover.
  • Costs are modeled as aggregate all-in round-trip approximations (commissions + slippage proxies), not measured spreads or order-book impact.
• Key robustness features:
  • Deterministic simulators and stored model snapshots (SHA-256 hashes) used for reproducible auditing of July re-evaluation.
  • Explicit disclosure of consumed vs. frozen intervals and search budgets; missing artifacts were not reconstructed with fallbacks.

Implications for AI Economics

• Predictive metrics ≠ economic value: High discrimination (ROC AUC) on rare events does not ensure positive policy returns. For AI economics, this underscores the need to evaluate end-to-end economic objectives (costs, execution, abstention, portfolio state) rather than classification metrics alone.
• Importance of cost-aware design: Small gross predictive edges (single-digit bps) are fragile. Realistic, conservative cost models (spreads, slippage, queue effects) and turnover penalties must be integrated during model selection and objective design.
• Chronology, provenance, and adaptive search matter economically: Large search budgets and reuse of consumed periods inflate selection risk. Research claiming prospective economic gains must (a) lock a frozen specification before broad search, (b) disclose search budgets, and (c) keep clear artifact provenance to reduce overclaiming.
• Evaluation protocol best practices for trading-related ML:
  • Report precision/recall and average precision for imbalanced rare-event labels (PR metrics are more informative than ROC AUC under severe class imbalance).
  • Use conservative execution assumptions: next-open vs same-close semantics, adverse tie-breaking, and multiple cost stress tests.
  • Separate evidence classes (artifact-backed prospective, model-specific later-period, consumed diagnostic) and avoid treating consumed diagnostics as fresh holdouts.
  • Apply abstention/coverage-aware policies (selective classification) and incorporate turnover control or policy-aware learning objectives (predict-then-optimize or cost-sensitive losses).
• Role of AI in the research lifecycle — helpful but bounded: Autonomous agents can accelerate literature retrieval, artifact checks, and reproducibility work, but they must be constrained away from live market decision-making. The audit shows AI-assisted workflows can improve evidence integrity without acting as trading agents.
• External validity limits: The paper’s negative results are exchange-, universe-, period-, label-, and simulator-specific. They do not prove that ML cannot be profitable in other crypto or market settings, but they do illustrate how fragile apparent gains can be when realistic frictions and honest auditing are applied.
• Policy for research transparency in AI economics: publish raw artifacts, simulation seeds/hashes, search logs, and clear chronological split documentation to enable independent evidence audits and reduce the risk of overfitting-driven false positives.

If you want, I can: - Extract key numerical tables and metrics into a concise CSV-ready summary. - Produce a short checklist for robust prediction-to-policy evaluation you can reuse in future AI economics studies.