Garbage in, garbage out: Why data quality dictates strategy survival
In quantitative finance, backtesting simulations are only as reliable as the empirical price series feeding them. A trading strategy does not operate in an abstract mathematical space; it consumes discrete historical records—Open, High, Low, Close (OHLC) bars, traded volume, and timestamps—to construct indicators, evaluate entry signals, and calculate execution prices.
When raw market data contains subtle corruptions, backtesting engines do not crash. Instead, they produce catastrophic research illusions:
- Phantom alpha: A bad exchange tick or corrupted low print creates an artificial -30% dip that instantly bounces back on the next bar. An automated mean-reversion algorithm buys the artificial bottom and books an impossible 30% gain in a single bar.
- Hidden execution failure: Unadjusted stock splits trigger massive false breakout signals or plunge trailing stops into liquidation.
- Temporal leakage: Duplicate or misordered timestamps scramble bar sequence causality, feeding future prints into indicator calculations.
- Timezone displacement: A 1-hour daylight saving shift or mixed UTC/local timestamps aligns the European market close with the wrong US trading session bar.
Corrupted Input Tape:
Time 09:30 ──> [Open: 100.0, High: 102.0, Low: 99.5, Close: 101.5] (Valid)
Time 09:31 ──> [Open: 101.5, High: 101.0, Low: 101.8, Close: 102.0] (LOGIC ERROR: High < Low!)
Time 09:31 ──> [Open: 101.5, High: 102.5, Low: 101.0, Close: 102.0] (DUPLICATE TIMESTAMP)
Time 09:35 ──> [Open: 102.0, High: 103.0, Low: 10.0, Close: 102.5] (BAD TICK: Low = 10.0!)
│
▼
Flawed Backtest Result: Sharpe 4.2, Max Drawdown 3.1% ──> LIVE CAPITAL LOSSTo prevent unexecutable simulations, quant research workflows must implement automated, deterministic data validation prior to running any backtest. Below is the 8-point data hygiene checklist every quantitative researcher must enforce on historical OHLCV data.
The 8-Point OHLC Data Hygiene Checklist
Before passing historical bars to an execution engine, evaluate the dataset against these eight foundational criteria.
┌────────────────────────────────────────────────────────┐
│ 8-POINT OHLC DATA HYGIENE AUDIT │
└────────────────────────────────────────────────────────┘
│
┌──────────────────┬───────────────────┼───────────────────┬──────────────────┐
▼ ▼ ▼ ▼ ▼
1. Timestamps 2. Price Bounds 3. Bar Logic 4. Missing Bars 5. Volume
- Monotonic - Strictly > 0 - High >= Max(O,C) - Calendar gaps - Finite >= 0
- Strict Unique - Finite (no Inf) - Low <= Min(O,C) - No blind ffill - Zero-volume rule
- No Duplicates - No NaN/null - High >= Low - Outlier halts
│ │
└──────────────────────────────┬──────────────────────────────────────────────┘
▼
┌─────────────────────────────┐
│ 6. Corporate Actions │ - Split vs Dividend adj
│ 7. Timezone Normalization │ - Enforce UTC / offsets
│ 8. Trades vs Quotes │ - Bid-ask bounce check
└─────────────────────────────┘1. Monotonic Timestamps: Strictly ascending DatetimeIndex without duplicates
A historical time series must establish an unambiguous, strictly increasing chronological order:
t_0 < t_1 < t_2 < \dots < t_NIf duplicate timestamps exist (t_i = t_{i+1}), order execution engines cannot determine which bar occurred first. This leads to non-deterministic fills and corrupted indicator state. Common causes include:
- Multi-exchange data dumps where two venue feeds are concatenated without deduplication.
- Daylight Saving Time (DST) fall-back transitions in local time zones (e.g., 01:30 AM occurs twice).
- Truncated millisecond precision when saving high-frequency data to CSV files.
Audit Rule: The DataFrame index must be a pd.DatetimeIndex, with df.index.is_monotonic_increasing == True and df.index.is_unique == True.
2. Price Bounds: Zero, negative, or infinite values
For equities, currencies, and cryptocurrencies, nominal asset prices must be strictly positive and finite:
P \in (0, \infty) \quad \forall P \in \{\text{Open}, \text{High}, \text{Low}, \text{Close}\}While certain energy futures physically settled into negative prices during extraordinary contango events (such as WTI crude oil in April 2020), standard single-asset spot models cannot handle zero or negative prices without breaking log returns, percentage drawdowns, and portfolio sizing equations:
R_t = \ln(P_t / P_{t-1}) \quad \text{is undefined if } P \le 0Furthermore, floating-point NaN, None, and Inf values propagate through rolling indicators (like SMA, EMA, and ATR), permanently corrupting all subsequent bars.
Audit Rule: Assert that all price elements are finite numbers strictly greater than zero ((df[cols] > 0).all() & np.isfinite(df[cols]).all()).
3. Bar Logical Integrity: High must be the supremum, Low the infimum
An OHLC bar represents the summarized price action within a fixed discrete interval [t_{\text{start}}, t_{\text{end}}). By definition:
- High is the maximum price traded during the bar:
\text{High}_t \ge \max(\text{Open}_t, \text{Close}_t) - Low is the minimum price traded during the bar:
\text{Low}_t \le \min(\text{Open}_t, \text{Close}_t) - Range:
\text{High}_t \ge \text{Low}_t > 0
Valid Bar Structure: Inverted Bar (Corrupted Data):
High (Supremum) Open (Above High!)
│ │
┌────┴────┐ ┌────┴────┐
│ Open │ │ High │
│ │ │ │
│ Close │ │ Close │
└────┬────┘ └────┬────┘
│ │
Low (Infimum) LowData scraping errors, unaligned column mappings (e.g., swapping Open and High), or flawed OTC trade aggregation frequently produce inverted bars where Low > High or Open > High. If fed into a backtester, limit orders will trigger at prices that never existed within exchange bounds.
4. Missing Bars and Weekend/Holiday gaps
Time gaps in historical data fall into two distinct categories:
- Normal calendar closures: Weekends, official exchange holidays (e.g., Christmas, Martin Luther King Jr. Day), and overnight non-trading hours in cash equity markets.
- Anomalous missing bars: Data vendor dropouts, dropped websocket packets, or unexpected regulatory trading halts (e.g., LULD halts or Code T1 pending news).
[!WARNING] Never blindly forward-fill (
df.ffill()) missing intraday bars without rigorous verification. Forward-filling prices during an unscheduled exchange halt creates synthetic bars with zero volatility. A naive backtest will assume continuous liquidity and fill limit orders that could never have traded.
Audit Rule: Compute the timestamp difference between adjacent bars:
\Delta t_k = t_k - t_{k-1}Flag all intervals where \Delta t_k exceeds the expected bar period, filter out documented market calendar closures, and isolate unexplained data holes.
5. Volume Verification: Non-negative and zero-volume handling
Traded volume represents the total quantity of shares or contracts exchanged during the bar. It must satisfy:
V_t \ge 0 \quad \text{and} \quad V_t < \inftyNegative volume indicates corrupted data feeds or reversed settlement ledger adjustments erroneously mixed into market tapes.
Furthermore, zero-volume bars (V_t = 0) require deliberate architectural handling:
- In 24/7 liquid markets (like major crypto pairs), a zero-volume bar indicates a network outage or exchange API downtime.
- In illiquid equities, pre-market trading, or forex off-peak hours, zero volume indicates that quotes moved without any matched trade. If a price change is recorded (
\text{Close}_t \ne \text{Close}_{t-1}) alongsideV_t = 0, the bar represents quote updates rather than executable liquidity.
6. Corporate Actions: Verifying split and dividend adjustments
When testing equity strategies over long horizons, unadjusted corporate actions introduce fatal distortions:
- Forward and reverse stock splits: A 4-for-1 split overnight reduces the nominal share price by 75% while multiplying shares by 4. If using unadjusted data, momentum indicators will register a massive false crash, triggering stop-losses and short signals.
- Cash dividends: Standard retail feeds supply an
Adj Closethat retroactively deflates historical prices across all corporate actions.
| Price Representation | Formula / Nature | Impact on Indicators | Impact on Fixed-Dollar Rules |
|---|---|---|---|
| Raw Unadjusted | Exact exchange tape prints | Distorted across split boundaries | Accurate to historical tape |
| Split-Adjusted | Prices scaled by split factor \prod f_{\text{split}} | Preserves chart geometry and volatility | Maintains realistic price scales |
| Total Return (Dividend-Adjusted) | Prices scaled by dividends and splits | Smooth compounding returns | Distorts dollar stops and penny filters |
[!TIP] Quant Engineering Rule: For single-asset price-action backtesting, use split-adjusted OHLCV data. Do not use dividend-adjusted prices for intraday or price-level execution rules, as dividend discounting artificially depresses prices from 10 years ago (e.g., turning a $50 stock into $4.50), causing absolute price filters (such as
Close > 5.0) to reject legitimate blue-chip stocks.
7. Timezone Normalization: Enforcing UTC across all series
Financial data feeds originate from global exchanges operating across diverse regional time zones:
- NYSE / Nasdaq:
America/New_York(UTC-5 in winter, UTC-4 in summer) - London Stock Exchange:
Europe/London(UTC+0 in winter, UTC+1 in summer) - Tokyo Stock Exchange:
Asia/Tokyo(UTC+9, no DST) - Crypto Exchanges:
UTC
Storing timestamps in naive (unlocalized) datetime formats leads to silent alignment errors during Daylight Saving Time switches. For example, when New York switches from Eastern Daylight Time (EDT) to Eastern Standard Time (EST) in November, local clocks repeat the 01:00 hour, creating duplicate local timestamps.
Audit Rule: All historical datasets must be stored with explicit timezone information, preferably localized and converted to UTC:
df.index = pd.to_datetime(df.index).tz_localize("UTC")
# Or if originally recorded in exchange time:
df.index = pd.to_datetime(df.index).tz_localize("America/New_York").tz_convert("UTC")8. Bid-Ask Spread vs. Mid-Price: Trades vs. Quotes
Most historical bar datasets record trade prints (last executed transactions). However, executing trades inside the bid-ask spread is physically impossible in real markets.
Ask Price: $100.05 ────┐
├── Bid-Ask Spread ($0.10)
Bid Price: $ 99.95 ────┘
Last Trade: $100.00 (Midpoint) <── Backtest assumes this price is fillable for both buys & sells!If your dataset represents trade prints:
- Consecutive trades alternating between the bid and ask create artificial high-frequency volatility ("bid-ask bounce").
- Mean-reversion backtests will profit from bounce that cannot be captured in live execution because market buy orders must cross the spread and fill at the ask (
100.05).
Audit Rule: Verify whether your data source provides trade OHLC, mid-price OHLC, or separate top-of-book (Bid/Ask) bars. When backtesting on trade or mid-price bars, apply realistic slippage models (e.g., 5 to 10 basis points) to account for crossing the spread.
Complete Python code: An automated data sanitization pipeline
The following self-contained Python script implements a production-ready validation pipeline. It audits historical OHLCV DataFrames against all 8 checklist items, detects anomalies, and generates an audit report with actionable diagnostic metrics.
"""
Production OHLCV Data Sanitization and Validation Pipeline.
Validates historical market data against the 8-Point Data Hygiene Checklist.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
import numpy as np
import pandas as pd
@dataclass
class ValidationReport:
"""Detailed audit report produced by OHLCV validation."""
is_valid: bool
total_bars: int
start_time: pd.Timestamp | None
end_time: pd.Timestamp | None
issues_found: int
errors: list[str] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
metrics: dict[str, Any] = field(default_factory=dict)
def summary(self) -> str:
"""Render a formatted text audit summary."""
status = "PASSED" if self.is_valid else "FAILED"
lines = [
"=" * 60,
f"OHLCV DATA VALIDATION AUDIT: {status}",
"=" * 60,
f"Total Bars Scanned : {self.total_bars:,}",
f"Date Range : {self.start_time} to {self.end_time}",
f"Issues Encountered : {self.issues_found}",
"-" * 60,
]
if self.errors:
lines.append("CRITICAL ERRORS:")
for err in self.errors:
lines.append(f" [X] {err}")
if self.warnings:
lines.append("WARNINGS:")
for warn in self.warnings:
lines.append(f" [!] {warn}")
if not self.errors and not self.warnings:
lines.append(" [✓] All 8 hygiene checklist items verified successfully.")
lines.append("=" * 60)
return "\n".join(lines)
def validate_ohlc(
df: pd.DataFrame,
expected_freq: str | None = None,
allow_zero_volume: bool = True,
) -> tuple[bool, ValidationReport]:
"""
Validate historical OHLCV data against the 8-point hygiene checklist.
Parameters
----------
df : pd.DataFrame
Historical price data with columns 'Open', 'High', 'Low', 'Close',
and optionally 'Volume'.
expected_freq : str, optional
Expected pandas offset string (e.g., '1D', '1h', '5min') to verify
calendar continuity.
allow_zero_volume : bool, default True
If False, bars with Volume == 0 trigger warnings.
Returns
-------
tuple[bool, ValidationReport]
Boolean pass/fail flag and comprehensive audit report.
"""
errors: list[str] = []
warnings: list[str] = []
metrics: dict[str, Any] = {}
# Check 0: DataFrame existence and row count
if df.empty:
report = ValidationReport(
is_valid=False,
total_bars=0,
start_time=None,
end_time=None,
issues_found=1,
errors=["DataFrame is empty."],
)
return False, report
total_bars = len(df)
required_cols = ["Open", "High", "Low", "Close"]
missing_cols = [c for c in required_cols if c not in df.columns]
if missing_cols:
errors.append(f"Missing essential price columns: {missing_cols}")
report = ValidationReport(
is_valid=False,
total_bars=total_bars,
start_time=None,
end_time=None,
issues_found=len(errors),
errors=errors,
)
return False, report
# 1. Monotonic Timestamps & Uniqueness
if not isinstance(df.index, pd.DatetimeIndex):
errors.append("Index must be a pandas DatetimeIndex.")
else:
if not df.index.is_monotonic_increasing:
errors.append("Timestamps are not strictly ascending (is_monotonic_increasing=False).")
if not df.index.is_unique:
dup_count = df.index.duplicated().sum()
errors.append(f"Found {dup_count} duplicate timestamp(s) in index.")
# 7. Timezone Normalization
if df.index.tz is None:
warnings.append("Index is naive (no timezone). Localize to UTC or exchange timezone.")
else:
metrics["timezone"] = str(df.index.tz)
# 2. Price Bounds: Finite, Positive Values
for col in required_cols:
series = df[col]
# Null / NaN check
null_count = series.isna().sum()
if null_count > 0:
errors.append(f"Column '{col}' contains {null_count} NaN/null values.")
# Infinite values
inf_count = np.isinf(series).sum()
if inf_count > 0:
errors.append(f"Column '{col}' contains {inf_count} Infinite values.")
# Non-positive prices (<= 0)
non_positive = (series <= 0).sum()
if non_positive > 0:
errors.append(f"Column '{col}' contains {non_positive} zero or negative values.")
# 3. Bar Logical Integrity
high = df["High"]
low = df["Low"]
open_p = df["Open"]
close = df["Close"]
# High < Low check
inv_range = (high < low).sum()
if inv_range > 0:
errors.append(f"Found {inv_range} bar(s) where High < Low.")
# High >= max(Open, Close)
max_oc = np.maximum(open_p, close)
high_violations = (high < max_oc).sum()
if high_violations > 0:
errors.append(f"Found {high_violations} bar(s) where High < max(Open, Close).")
# Low <= min(Open, Close)
min_oc = np.minimum(open_p, close)
low_violations = (low > min_oc).sum()
if low_violations > 0:
errors.append(f"Found {low_violations} bar(s) where Low > min(Open, Close).")
# 4. Missing Bars and Frequency Checks
if isinstance(df.index, pd.DatetimeIndex) and len(df) > 1 and expected_freq:
time_deltas = df.index.to_series().diff().dropna()
expected_td = pd.Timedelta(expected_freq)
anomalous_gaps = (time_deltas > expected_td).sum()
if anomalous_gaps > 0:
max_gap = time_deltas.max()
warnings.append(
f"Found {anomalous_gaps} timestamp gap(s) exceeding {expected_freq}. "
f"Max gap: {max_gap}."
)
metrics["median_bar_delta"] = str(time_deltas.median())
# 5. Volume Verification
if "Volume" in df.columns:
vol = df["Volume"]
vol_nulls = vol.isna().sum()
if vol_nulls > 0:
errors.append(f"Column 'Volume' contains {vol_nulls} NaN values.")
vol_neg = (vol < 0).sum()
if vol_neg > 0:
errors.append(f"Column 'Volume' contains {vol_neg} negative values.")
vol_zero = (vol == 0).sum()
metrics["zero_volume_bars"] = int(vol_zero)
if vol_zero > 0 and not allow_zero_volume:
warnings.append(f"Column 'Volume' contains {vol_zero} zero-volume bar(s).")
else:
warnings.append("Optional 'Volume' column not provided.")
# 6. Corporate Action Split Anomaly Heuristic
# Flag single-day unadjusted price drops > 40%
pct_changes = close.pct_change().dropna()
suspicious_drops = pct_changes[pct_changes < -0.40]
if not suspicious_drops.empty:
warnings.append(
f"Detected {len(suspicious_drops)} single-bar drop(s) > 40%. "
f"Verify if unadjusted stock splits exist (e.g. at {suspicious_drops.index[0]})."
)
# 8. Spread / Volatility Sanity Check
bar_spreads = (high - low) / close
zero_spread_bars = (bar_spreads == 0).sum()
metrics["zero_spread_bars"] = int(zero_spread_bars)
if zero_spread_bars > (total_bars * 0.05):
warnings.append(
f"{zero_spread_bars} bar(s) ({zero_spread_bars/total_bars:.1%}) have zero range "
"(High == Low). Verify if feed represents stale prices or illiquid periods."
)
is_valid = len(errors) == 0
start_ts = df.index[0] if isinstance(df.index, pd.DatetimeIndex) and len(df) else None
end_ts = df.index[-1] if isinstance(df.index, pd.DatetimeIndex) and len(df) else None
report = ValidationReport(
is_valid=is_valid,
total_bars=total_bars,
start_time=start_ts,
end_time=end_ts,
issues_found=len(errors) + len(warnings),
errors=errors,
warnings=warnings,
metrics=metrics,
)
return is_valid, report
if __name__ == "__main__":
# Demonstrate pipeline on synthetic data with deliberate corruptions
dates = pd.date_range("2026-01-01", periods=10, freq="1D", tz="UTC")
# Construct test dataset with deliberate anomalies
raw_data = {
"Open": [100.0, 102.0, 101.0, 99.0, 105.0, 104.0, 103.0, 105.0, 106.0, 108.0],
"High": [103.0, 104.0, 100.0, 102.0, 106.0, 107.0, 105.0, 108.0, 109.0, 110.0], # Bar 2: High < Open!
"Low": [ 99.0, 100.0, 101.5, 98.0, 103.0, 102.0, 101.0, 104.0, 105.0, 106.0], # Bar 2: Low > High!
"Close": [102.0, 101.0, 99.5, 101.0, 104.0, 105.0, 104.0, 107.0, 108.0, 109.0],
"Volume": [1000, 1500, 1200, 800, -500, 2000, 1800, 0, 2200, 1900], # Bar 4: Negative vol!
}
sample_df = pd.DataFrame(raw_data, index=dates)
is_clean, audit = validate_ohlc(sample_df, expected_freq="1D")
print(audit.summary())When executed against corrupted data, the script captures every integrity breach:
============================================================
OHLCV DATA VALIDATION AUDIT: FAILED
============================================================
Total Bars Scanned : 10
Date Range : 2026-01-01 00:00:00+00:00 to 2026-01-10 00:00:00+00:00
Issues Encountered : 4
------------------------------------------------------------
CRITICAL ERRORS:
[X] Found 1 bar(s) where High < Low.
[X] Found 1 bar(s) where High < max(Open, Close).
[X] Found 1 bar(s) where Low > min(Open, Close).
[X] Column 'Volume' contains 1 negative values.
============================================================How OpenBacktesting enforces data contracts
OpenBacktesting (version 0.2.1) is designed specifically for single-asset historical bar-by-bar research. Rather than making silent assumptions or applying hidden transformations, OpenBacktesting enforces strict architectural data contracts at the engine boundary.
1. The Single-Asset Ingestion Contract
OpenBacktesting accepts a standard pandas DataFrame representing one asset's chronological tape. The input requirements are strictly validated:
- Required columns: Exactly
Open,High,Low,Close, and optionalVolume. - Finite, positive prices: Missing (
NaN), infinite, or negative price points are rejected immediately because orders cannot match against corrupt prints. - Strictly ascending, unique index: Every row must represent one discrete bar in chronological sequence.
import pandas as pd
from openbacktesting import Backtest, Strategy, DataContext, Timeframe
# 1. Load historical CSV data
df = pd.read_csv("clean_aapl_daily.csv", parse_dates=["Date"], index_col="Date")
# 2. Enforce chronological sort and timezone
df = df.sort_index()
if df.index.tz is None:
df.index = df.index.tz_localize("America/New_York")
# 3. Restrict columns to exact contract specifications
df = df[["Open", "High", "Low", "Close", "Volume"]]2. Timeframe and Timezone Specification via DataContext
In OpenBacktesting, temporal metadata is declared explicitly through DataContext:
# Declare daily timeframe and explicit timezone
context = DataContext(timeframe=Timeframe.days(1), timezone="America/New_York")The role of DataContext highlights an essential quant design principle:
[!IMPORTANT]
DataContextconverts display timezones while preserving exact physical instants; it does NOT automatically resample bars.
Many legacy frameworks attempt automatic hidden resampling (e.g., converting 1-hour input bars into daily bars behind the scenes). Silent resampling is dangerous because it masks incomplete sessions, aggregates unclosed bars into premature signals, and introduces subtle lookahead bias. In OpenBacktesting, data must be cleaned and aggregated to its target timeframe before entering the engine, guaranteeing that what the strategy sees in backtesting matches what it would observe bar by bar in production.
3. Causal Data Access Inside Strategies
Inside an OpenBacktesting strategy, data series are exposed as read-only cursor handles:
class DataHygieneStrategy(Strategy):
def init(self):
# self.close provides a dynamic handle tracking the current bar
self.sma20 = self.ta.sma(self.close, length=20)
def next(self):
# Ensure indicators have warmed up before reading
if not self.sma20.valid[-1]:
return
# Current bar's close
current_close = self.close[-1]
# Previous bar's close (requires at least 2 bars)
if len(self.close) >= 2:
prev_close = self.close[-2]
# Current bar's timestamp
current_time = self.data_index[-1]- In
init(), the entire historical series is visible for precomputing vectorized indicator buffers. - In
next(), reads are causally bounded toself.close[-1](the current bar) and historical bars (self.close[-2]). You cannot peek intoself.close[0]of tomorrow. - Indicators provide
.validdiagnostics so strategies never trade on uninitializedNaNstates.
Summary checklist for quant data pipelines
Before running a backtest or optimizing strategy parameters, execute this five-step pre-flight checklist on your data ingestion pipeline:
Quant Data Pipeline Pre-Flight Checklist:
[ ] 1. Schema & Types: DataFrame has 'Open', 'High', 'Low', 'Close' as float64; index is DatetimeIndex.
[ ] 2. Strict Ordering: df.index.is_monotonic_increasing is True; duplicates eliminated.
[ ] 3. Price Integrity: All prices > 0, finite; High >= max(Open, Close); Low <= min(Open, Close).
[ ] 4. Timezone Anchor: Timestamps explicitly localized (UTC recommended) to eliminate DST shifts.
[ ] 5. Split Verification: Prices are split-adjusted to preserve chart continuity; cash dividends decoupled.By enforcing automated data hygiene before simulation, you ensure that backtest metrics reflect legitimate market mechanics rather than data vendor artifacts.
Further Reading & Related Guides
- What Is Lookahead Bias? How to Prevent Future Data Leaks in Python Backtests — Learn how causal bar indices and shifted signals protect backtests from peeking into future prices.
- Stock Backtesting in Python: Data Adjustments, Costs and Benchmarks — A comprehensive guide to modeling equity corporate actions, borrow costs, and buy-and-hold baselines.
- OpenBacktesting Data Reference Documentation — Detailed API reference for
DataContext,Timeframe, and data contract validation rules.
