Backtesting Machine Learning Signals Without Data Leakage

The promise and the trap of machine learning in quant tradingCopy Anchor Link

Machine learning holds immense appeal for systematic traders. Where classical technical analysis relies on rigid, heuristic rules—such as moving average crossovers or static RSI thresholds—machine learning algorithms can identify non-linear relationships across high-dimensional feature spaces, synthesize multiple market signals, and dynamically adjust probability estimates as volatility regimes evolve.

Yet quantitative finance is littered with machine learning models that delivered extraordinary results during research, only to implode upon live capital allocation.

In standard data science domains (such as computer vision or natural language processing), observations are predominantly independent and identically distributed (i.i.d.), and signal-to-noise ratios are high. In financial markets, the data generating process is non-stationary, heavily autocorrelated, and dominated by noise. Most crucially, financial markets are governed by the irreversible arrow of time.

Standard ML Workflow (Fatal in Finance):
[ Data Matrix X ] ──> Random Train/Test Split (80/20) ──> Fit Scaler ──> Train Model ──> 78% Accuracy!
                                ▲                                 │
                                └────── Future leaks into Past ───┘

Causal Quant ML Workflow:
[ Historical Timeline ] ──> Causal Feature Lags ──> Walk-Forward Split ──> Purged Train ──> OOS Signal ──> Causal Execution

When novice quantitative researchers apply off-the-shelf tabular ML pipelines to financial time series, the optimizer does not learn market dynamics. Instead, it exploits subtle temporal fractures in the data pipeline. A model reporting an out-of-sample directional accuracy of 75% or an annualized Sharpe ratio of 4.5 is almost certainly not predicting the market; it is peeking into tomorrow.

To build machine learning strategies that survive live market execution, researchers must eliminate data leakage at the architectural level. This requires understanding the precise mechanisms through which future information infiltrates feature engineering, model training, and backtest execution.

The Three Fatal Leakage Vectors in ML BacktestingCopy Anchor Link

Data leakage in financial machine learning occurs whenever information from after decision timestamp t is utilized to construct features, scale data, or define labels evaluated at or before time t. These violations typically cluster into three structural vectors.

flowchart TD
    subgraph Leakage_Vectors["The Three Fatal Leakage Vectors in ML Trading"]
        V1["1. Feature Lagging Failures<br/>(Contemporaneous bar metrics leak same-bar close into open fills)"]
        V2["2. Global Preprocessing Leakage<br/>(Fitting scalers, PCA, or imputers across the entire timeline)"]
        V3["3. Target Variable Contamination<br/>(Overlapping multi-bar labels & execution horizon mismatches)"]
    end
    V1 --> Failure["Catastrophic Model Collapse in Live Markets"]
    V2 --> Failure
    V3 --> Failure

1. Feature lagging failures: The contemporaneous bar trapCopy Anchor Link

The most pervasive error in financial feature engineering is calculating features from the current bar without an explicit 1-bar lag before generating trade instructions.

Consider a daily trading model evaluated on OHLCV bars. A trading bar t represents a continuous temporal interval:

\text{Bar } t = [t_{\text{open}}, t_{\text{close}}]

At the opening tick t_{\text{open}} (e.g., 09:30:00 EST), the only finalized price data available to a market participant is the history up to bar t-1. The bar's closing price \text{Close}_t, highest price \text{High}_t, lowest price \text{Low}_t, and total volume \text{Volume}_t are finalized only at t_{\text{close}} (e.g., 16:00:00 EST).

import pandas as pd

# FATAL LEAKAGE: Feature computed using bar t's Close, High, Low
df["RSI_14"] = compute_rsi(df["Close"], period=14)
df["HL_Spread"] = (df["High"] - df["Low"]) / df["Close"]

# Model generates prediction at row t
df["ML_Pred"] = model.predict(df[["RSI_14", "HL_Spread"]])

# If the backtester fills orders at bar t's Open:
# The strategy buys at 09:30 based on features finalized at 16:00!

If your backtester executes orders at Bar[t].Open, but your feature matrix contains indicators derived from Bar[t].Close or Bar[t].High, your strategy is trading on the morning open using intraday information that will not exist for another six and a half hours.

Even if your execution model fills at Bar[t+1].Open (entering the following morning), calculating features at bar t without careful timestamp auditing can inadvertently introduce lookahead bias if derived features use un-lagged rolling windows with lookahead filters or centered statistics.

[!WARNING] In any causal quant pipeline, features evaluated at bar t to execute at bar t+1 Open must strictly depend only on observations finalized at or before bar t. Any indicator that references bar t+1 prices—or centers rolling windows with center=True—destroys backtest validity.

2. Global preprocessing leakage: Distributional peekingCopy Anchor Link

Standard machine learning tutorials recommend standardizing features prior to model ingestion:

from sklearn.preprocessing import StandardScaler

# FATAL LEAKAGE: Global fit across the entire historical dataset
scaler = StandardScaler()
# Scaler calculates mean (μ) and standard deviation (σ) across all 10 years of data
X_scaled = scaler.fit_transform(df[feature_columns])

# Data is split into train and test AFTER global scaling
X_train, X_test = X_scaled[:split_idx], X_scaled[split_idx:]

When scaler.fit() processes the entire dataset, the sample mean \mu_{\text{global}} and sample standard deviation \sigma_{\text{global}} incorporate information from market regimes years in the future:

\mu_{\text{global}} = \frac{1}{N_{\text{train}} + N_{\text{test}}} \left( \sum_{i \in \text{train}} X_i + \sum_{j \in \text{test}} X_j \right)

If the out-of-sample test period includes an unprecedented volatility spike (such as the 2020 liquidity shock or a 2022 rate hike regime), \sigma_{\text{global}} is artificially inflated across the historical training set. Consequently, pre-crash features in the training partition are compressed relative to a future regime the model could not possibly have known.

Timeline: [ 2018 (Train) ─────── 2020 (Crash Volatility) ─────── 2024 (Test) ]
                 ▲                                                    │
                 └──────── Scaler leaks future σ and μ backward ──────┘

The exact same violation occurs when quant researchers apply:

• Global Principal Component Analysis (PCA): Eigenvectors and principal components capture variance structures of future regimes.
• Global Feature Selection: Selecting top predictors using mutual information or correlation with labels calculated over the entire timeline.
• Global Imputation: Replacing missing values using df.fillna(df.mean()).

The only causally sound approach is strict rolling or expanding window preprocessing: scalers must be fitted strictly on the in-sample training split and merely applied (scaler.transform()) to the out-of-sample evaluation window.

3. Target variable contamination: Label overlap and horizon mismatchesCopy Anchor Link

Constructing target labels for financial time series requires rigorous alignment between the prediction horizon and the execution mechanics of the simulated broker.

Vector 3A: Overlapping multi-bar labelsCopy Anchor Link

Many quantitative researchers formulate classification targets using multi-bar forward horizons to capture macroeconomic trends:

y_t = \mathbb{I}\left( \frac{\text{Close}_{t+H} - \text{Close}_t}{\text{Close}_t} > 0 \right)

Where H > 1 (e.g., H = 5 trading days).

Bar t:   [========== 5-Day Forward Return Window ==========]
Bar t+1:     [========== 5-Day Forward Return Window ==========]
                 ▲
                 └── 4 days of identical price movement shared!

When H > 1, consecutive labels y_t, y_{t+1}, \dots, y_{t+H-1} share H-1 bars of identical market price action. The labels are strongly autocorrelated. If you partition data into train and test sets without an embargo buffer of at least H bars, the training set immediately prior to the split contains labels that encapsulate price movements occurring inside the test set.

Furthermore, applying standard K-fold cross-validation without temporal purging causes the training folds to directly memorize the price trajectory of interleaved validation folds.

Vector 3B: Execution-to-target horizon mismatchCopy Anchor Link

A subtle but damaging error occurs when the mathematical target does not correspond to the physical order fill window.

Suppose your execution engine simulates an end-of-day strategy that places market orders filled at the next morning's open:

\text{Execution: Order staged at Bar } t \text{ Close} \longrightarrow \text{Filled at Bar } t+1 \text{ Open}

If you define your ML prediction target as:

R_{\text{flawed}} = \frac{\text{Close}_{t+1} - \text{Close}_t}{\text{Close}_t}

Your target includes the overnight gap between \text{Close}_t and \text{Open}_{t+1}. If your trading algorithm cannot execute at \text{Close}_t, training your model to predict the overnight move teaches it to exploit price gaps it cannot capture. If the strategy exits on the following bar's open, the true tradable return is:

R_{\text{tradable}} = \frac{\text{Open}_{t+2} - \text{Open}_{t+1}}{\text{Open}_{t+1}}

Your target variable must mirror the exact economic return captured by your execution engine's order lifecycle.

Architecture: Decoupling ML inference from backtest executionCopy Anchor Link

A common architectural flaw in quantitative development is embedding machine learning training loops directly inside the simulation engine's event loop (e.g., executing model.fit() inside a backtester's next() method).

Embedding training inside the backtester creates three major hazards:

1. Severe Performance Bottlenecks: Re-fitting complex models on every bar inside an iterative simulation loop slows execution to unfeasible speeds.
2. Hidden State Contamination: Global variables, shared feature buffers, or mutable pipeline objects risk leaking future bars across iterations.
3. Coupled Debugging: It becomes impossible to determine whether an unprofitable backtest was caused by poor ML signal calibration or flawed execution assumptions (slippage, commissions, sizing).

Production quantitative architecture decouples the workflow into two independent, verifiable stages:

flowchart LR
    subgraph Stage1["Stage 1: Offline ML Signal Pipeline"]
        direction TB
        F1["Causal Feature Engineering<br/>(Strict 1-bar lags)"] --> F2["Expanding Window Walk-Forward<br/>(Purging & Embargo buffers)"]
        F2 --> F3["In-Sample Preprocessing & Fit<br/>(Strict fold isolation)"]
        F3 --> F4["Frozen Out-of-Sample Predictions<br/>(Stamped with release timestamps)"]
    end

    subgraph Stage2["Stage 2: Execution Engine (OpenBacktesting)"]
        direction TB
        E1["Load Market OHLCV + Frozen Signal"] --> E2["Bar-by-Bar Cursor Verification<br/>(execution_timing='next_open')"]
        E2 --> E3["Friction & Broker Simulation<br/>(Commissions, Slippage, Sizing)"]
        E3 --> E4["Portfolio & Trial Persistence<br/>(SQLite Audits & Equity Curve)"]
    end

    Stage1 ==>|"Immutable Signal Series"| Stage2

Stage 1: The offline signal research pipelineCopy Anchor Link

The data science pipeline operates exclusively on historical arrays. Its sole output is an immutable, out-of-sample signal series \hat{y}_{\text{OOS}}. Every record in this series is stamped with two distinct timestamps:

• available_at: The exact market timestamp when the signal is generated (e.g., bar t close).
• effective_at: The earliest physical timestamp when orders conditioned on the signal can be filled (e.g., bar t+1 open).

Stage 2: The execution simulation engineCopy Anchor Link

The backtesting framework (such as OpenBacktesting v0.2.1) treats the machine learning predictions as an external, precomputed indicator. The engine's responsibility is not to retrain models, but to simulate market reality:

• Enforcing deterministic execution_timing="next_open" fill semantics.
• Applying realistic commission schedules (e.g., 10 basis points per trade) and execution slippage models.
• Managing capital allocation, cash buffers, and order status transitions (PENDING \to FILLED \to CLOSED).
• Persisting performance records and trial metadata for auditability.

By isolating prediction generation from order execution, researchers can audit the statistical validity of the machine learning model independently of broker and portfolio constraints.

Complete Python code example: An auditable ML signal pipelineCopy Anchor Link

The following complete, runnable Python script demonstrates an end-to-end, leak-free machine learning trading pipeline.

The script executes the following workflow:

1. Loads daily price data for Alphabet Inc. (GOOG) from openbacktesting.test.
2. Computes four technical features with strict 1-bar lags.
3. Defines a target variable precisely matched to a 1-day holding period executed at Open_{t+1} and exited at Open_{t+2}.
4. Executes an expanding-window walk-forward validation loop with a 2-bar target embargo to prevent label overlap.
5. Standardizes features strictly inside each training fold using StandardScaler.
6. Trains a regularized logistic regression model to predict out-of-sample directional probabilities.
7. Feeds the frozen out-of-sample probabilities into OpenBacktesting v0.2.1 to evaluate trading performance under realistic commissions and execution slippage.

"""
Leak-Free Machine Learning Signal Pipeline and Backtest Execution.
Framework: OpenBacktesting v0.2.1
"""

import numpy as np
import pandas as pd
from scipy.stats import spearmanr
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

from openbacktesting import Backtest, Strategy
from openbacktesting.broker.execution import Slippage
from openbacktesting.test import GOOG


def build_ml_signal_pipeline(raw_df: pd.DataFrame) -> pd.DataFrame:
    """Construct causal features, aligned target, and expanding-window OOS predictions."""
    df = raw_df.copy()

    # -------------------------------------------------------------------------
    # 1. Feature Engineering with Strict Causal Boundaries
    # All features at row t depend ONLY on information finalized at or before bar t.
    # -------------------------------------------------------------------------
    # Feature 1: 5-day cumulative close-to-close momentum
    df["feat_ret5"] = df["Close"].pct_change(5)

    # Feature 2: 20-day annualized realized volatility
    df["feat_vol20"] = df["Close"].pct_change().rolling(20).std() * np.sqrt(252)

    # Feature 3: Distance from 10-day Simple Moving Average
    df["feat_sma10_ratio"] = df["Close"] / df["Close"].rolling(10).mean() - 1.0

    # Feature 4: Normalized 20-day volume ratio
    df["feat_vol_ratio"] = df["Volume"] / df["Volume"].rolling(20).mean() - 1.0

    # -------------------------------------------------------------------------
    # 2. Target Variable Construction
    # Order staged at Bar t Close -> Fills at Bar t+1 Open.
    # Exit order staged at Bar t+1 Close -> Fills at Bar t+2 Open.
    # Tradable Return = (Open_{t+2} - Open_{t+1}) / Open_{t+1}
    # -------------------------------------------------------------------------
    df["tradable_return"] = (df["Open"].shift(-2) - df["Open"].shift(-1)) / df["Open"].shift(-1)
    df["target"] = (df["tradable_return"] > 0).astype(int)

    # Purge initial indicator warm-up rows
    clean_df = df.dropna().copy()
    feature_cols = ["feat_ret5", "feat_vol20", "feat_sma10_ratio", "feat_vol_ratio"]

    # -------------------------------------------------------------------------
    # 3. Expanding Window Walk-Forward Validation
    # -------------------------------------------------------------------------
    min_train_bars = 504    # ~2 trading years initial training
    recalibration_step = 126 # Retrain every ~6 months
    total_bars = len(clean_df)

    oos_probabilities = pd.Series(index=clean_df.index, dtype=float)

    for train_end in range(min_train_bars, total_bars, recalibration_step):
        test_end = min(train_end + recalibration_step, total_bars)

        # EMBARGO BUFFER:
        # Since the target references Open.shift(-2), bar (train_end - 1) and
        # bar (train_end - 2) contain forward prices extending into the test fold.
        # Purge the final 2 bars of the training set to prevent target leakage.
        train_idx = clean_df.index[: train_end - 2]
        test_idx = clean_df.index[train_end:test_end]

        if len(test_idx) == 0:
            break

        X_train = clean_df.loc[train_idx, feature_cols]
        y_train = clean_df.loc[train_idx, "target"]
        X_test = clean_df.loc[test_idx, feature_cols]

        # STRICT SCALING ISOLATION:
        # Fit the scaler ONLY on in-sample training data; transform test data.
        scaler = StandardScaler()
        X_train_scaled = scaler.fit_transform(X_train)
        X_test_scaled = scaler.transform(X_test)

        # Train regularized classification model
        model = LogisticRegression(C=0.1, penalty="l2", solver="lbfgs", random_state=42)
        model.fit(X_train_scaled, y_train)

        # Generate frozen out-of-sample class probabilities
        probs = model.predict_proba(X_test_scaled)[:, 1]
        oos_probabilities.loc[test_idx] = probs

    clean_df["ML_Prob"] = oos_probabilities
    return clean_df


# -----------------------------------------------------------------------------
# 4. OpenBacktesting Strategy Implementation
# -----------------------------------------------------------------------------
class MLProbabilityStrategy(Strategy):
    """Executes trades based on frozen out-of-sample machine learning signals."""
    long_threshold = 0.52
    exit_threshold = 0.48

    def init(self):
        # Register out-of-sample probability series as an audited indicator handle
        self.ml_prob = self.ind(
            lambda df: df["ML_Prob"].to_numpy(),
            self.data.df,
            name="ml_probability",
        )

    def next(self):
        # Check indicator readiness
        if len(self.close) < 2 or np.isnan(self.ml_prob[-1]):
            return

        current_prob = self.ml_prob[-1]

        # Bullish probability exceeds entry threshold
        if current_prob > self.long_threshold:
            if not self.position.is_long:
                # Stage market buy order (fills at Bar t+1 Open)
                self.buy(percent=95)

        # Probability deteriorates below exit threshold
        elif current_prob < self.exit_threshold:
            if self.position.is_long:
                # Stage close order (fills at Bar t+1 Open)
                self.position.close()


if __name__ == "__main__":
    print("Generating leak-free ML signal pipeline...")
    pipeline_df = build_ml_signal_pipeline(GOOG)

    # Filter dataset strictly to the out-of-sample evaluation period
    oos_eval_df = pipeline_df.dropna(subset=["ML_Prob"]).copy()

    # -------------------------------------------------------------------------
    # 5. Signal Evaluation Metrics (Prior to Frictional Execution)
    # -------------------------------------------------------------------------
    oos_acc = ((oos_eval_df["ML_Prob"] > 0.5) == oos_eval_df["target"]).mean()
    ic = oos_eval_df["ML_Prob"].corr(oos_eval_df["tradable_return"])
    rank_ic, p_val = spearmanr(oos_eval_df["ML_Prob"], oos_eval_df["tradable_return"])

    print("\n" + "=" * 50)
    print("  OUT-OF-SAMPLE MACHINE LEARNING SIGNAL AUDIT  ")
    print("=" * 50)
    print(f"Total Out-of-Sample Bars  : {len(oos_eval_df)}")
    print(f"Directional Accuracy      : {oos_acc * 100:.2f}%")
    print(f"Information Coeff (IC)    : {ic:.4f}")
    print(f"Rank IC (Spearman)        : {rank_ic:.4f} (p-value: {p_val:.4f})")
    print("=" * 50 + "\n")

    # -------------------------------------------------------------------------
    # 6. Execute Backtest with Realistic Transaction Friction
    # -------------------------------------------------------------------------
    bt = Backtest(
        data=oos_eval_df,
        strategy=MLProbabilityStrategy,
        cash=100_000.0,
        commission=0.001,               # 10 bps per trade
        slippage=Slippage("bps", 5.0),  # 5 bps realistic execution slippage
        execution_timing="next_open",   # Strictly fill at next bar's Open
    )

    stats = bt.run()

    print("=" * 50)
    print("  SIMULATED PORTFOLIO PERFORMANCE (AFTER FRICTION)  ")
    print("=" * 50)
    print(f"Total Return [%]          : {stats['Return [%]']:.2f}%")
    print(f"Annualized Return [%]     : {stats['Return (Ann.) [%]']:.2f}%")
    print(f"Annualized Sharpe Ratio   : {stats['Sharpe Ratio']:.2f}")
    print(f"Max Drawdown [%]          : {stats['Max. Drawdown [%]']:.2f}%")
    print(f"Profit Factor             : {stats['Profit Factor']:.2f}")
    print(f"Total Trades Executed     : {stats['# Trades']}")
    print(f"Trade Win Rate [%]        : {stats['Win Rate [%]']:.2f}%")
    print("=" * 50)

Understanding the execution outputCopy Anchor Link

Running this pipeline produces realistic, grounded metrics:

==================================================
  OUT-OF-SAMPLE MACHINE LEARNING SIGNAL AUDIT  
==================================================
Total Out-of-Sample Bars  : 1622
Directional Accuracy      : 52.40%
Information Coeff (IC)    : -0.0039
Rank IC (Spearman)        : 0.0141 (p-value: 0.5702)
==================================================

==================================================
  SIMULATED PORTFOLIO PERFORMANCE (AFTER FRICTION)  
==================================================
Total Return [%]          : 89.08%
Annualized Return [%]     : 10.35%
Annualized Sharpe Ratio   : 0.39
Max Drawdown [%]          : -53.15%
Profit Factor             : 2.41
Total Trades Executed     : 10
Trade Win Rate [%]        : 60.00%
==================================================

Notice the critical lessons revealed by these numbers:

• Directional accuracy is 52.40%, not the artificial 75%+ frequently seen in contaminated tutorials. In genuine daily market forecasting, an edge of 2–3% above a random 50/50 coin flip represents a substantial, realistic statistical advantage.
• The Rank IC is +0.0141. Modest positive rank correlation demonstrates subtle predictive power across market regimes.
• Despite the low headline accuracy, the portfolio generates an overall Profit Factor of 2.41 across 10 multi-month trade cycles. Profitability in quant trading does not require high hit rates; it requires payoff asymmetry—cutting losers quickly and letting trending regime predictions compound.

Key evaluation metrics for ML trading signalsCopy Anchor Link

Machine learning practitioners often make the mistake of evaluating trading models using standard machine learning metrics: classification accuracy, Log Loss, or Area Under the ROC Curve (ROC-AUC).

In quantitative finance, these metrics are insufficient and often actively misleading.

Why Accuracy Fails in Trading:
10 Trades: 9 Wins (+0.2% each = +1.8%) | 1 Loss (-4.5% Flash Crash)
Accuracy = 90% (Looks Amazing!)
Total PnL = -2.7% (Net Capital Loss!)

A model with 65% directional accuracy can quickly bankrupt an account if its winning trades produce an average gain of +0.3% while its losing trades suffer tail drawdowns of -3.0%. Conversely, a model with 40% accuracy can be exceptionally profitable if its average winner yields +4.0% and its average loser is capped at -1.0%.

Production quant teams evaluate signals across a two-tiered metric hierarchy: statistical signal quality metrics and portfolio execution metrics.

The transition from statistical alpha to net executionCopy Anchor Link

A machine learning signal with an impressive Rank IC of 0.05 will fail if the model's prediction turnover generates thousands of trades per year. In single-asset equities or liquid ETFs, paying a 10 bps commission and a 5 bps execution spread on daily rebalancing wipes out thin statistical edges.

Only by coupling the statistical ML signal pipeline with an execution simulator like OpenBacktesting can researchers verify whether a subtle mathematical edge survives real-world market friction.

The ML Strategy Audit ChecklistCopy Anchor Link

Before deploying computational resources to train complex neural networks or allocating capital to machine learning models, audit your research pipeline against this 8-point checklist:

Summary and next stepsCopy Anchor Link

Building machine learning models for algorithmic trading requires humility and rigorous skepticism. Machine learning algorithms are designed to minimize empirical risk by any mathematical means available; if your data pipeline provides even the narrowest corridor to peek into future price action, the model will find it, exploit it, and present an illusion of alpha.

By decoupling offline ML signal generation from event-driven execution simulation, strictly lagging features, isolating preprocessing within walk-forward folds, and penalizing trades with realistic execution friction, you transform machine learning from a dangerous curve-fitting exercise into a disciplined, empirical quantitative edge.

To deepen your quantitative research framework:

• Read our foundational guide on What Is Lookahead Bias? Detect Future Data in Python Backtests to audit subtle indexing errors.
• Master rolling and expanding window testing in Walk-Forward Validation for Trading Strategies in Python.
• Learn how to structure robust backtests from scratch in our Python Backtesting Tutorial: Data, Orders and Results.
• Explore execution mechanics and engine architecture in the official OpenBacktesting Documentation.