Backtest Overfitting and Locked Holdout Validation

Research context

Bailey, Borwein, López de Prado and co-authors describe backtest overfitting as trying many strategy variations on a limited dataset and selecting patterns that often fail on new data. Their Lawrence Berkeley Lab paper emphasizes multiple testing: a seemingly strong result becomes easier to find by chance as the number of attempts grows. That is why a trial count belongs beside a performance result.

The scikit-learn TimeSeriesSplit documentation warns that ordinary cross-validation can train on future data and evaluate on the past. Its expanding chronological folds and optional gap provide a general model for preserving time order. Trading research may need a larger purge based on indicator lookback, overlapping positions, labels, or operational delays; there is no universal gap size.

TradingView's official documentation also warns about lookahead, selection bias, overfitting, non-standard chart data, intrabar assumptions, and differences between historical and real-time calculation. A clean split cannot rescue a simulation whose rules use unavailable future information or unrealistic fills.

Backtest app setup example

Use an 18-month BTCUSDT one-hour dataset as a teaching example. Reserve the latest six months as a locked holdout, keep a seven-day gap before it, and use the earlier period for research. Predeclare a small family such as EMA fast lengths 8, 9, and 10 crossed with slow lengths 20, 21, and 22—nine trials, not an endless search.

For every trial, keep long direction, $10,000 capital, 2% position size, 2% stop, 4% target, 0.10% commission, 5 bps slippage, and closed-candle semantics fixed. Log failures as well as winners. Select one candidate using a rule written in advance that includes minimum trades, net result after costs, drawdown, outlier concentration, and parameter-neighborhood stability.

Save the candidate in Backtest with a rule fingerprint and screenshot. Do not change it after opening the holdout. Run the exact setup on the held-out six months and record the result once. A failed holdout is a valid rejection, not an invitation to optimize on those six months.

How to read the results

Compare research and holdout by mechanism, not by demanding identical returns. Ask whether trade direction, cadence, cost sensitivity, profit factor, drawdown, average R, and failure clusters remain plausible. Some degradation is expected; a sign reversal, disappearance of trades, or dependence on one outlier requires explanation and often rejection.

Report the number of tried configurations and the selection rule. A candidate chosen from nine attempts carries different selection risk from a candidate chosen from nine thousand hidden attempts. Include neighboring parameters so a narrow peak is visible.

Treat the holdout as a decision gate. Pass can mean advance to paper observation, not deploy capital. Fail means preserve the record and reformulate the hypothesis using a new research window. Inconclusive means collect later data rather than loosening the threshold after seeing the outcome.

What locked means in practice

A locked holdout has access controls in the research process even if the file is technically available. Record its date boundaries before tuning, avoid previewing its chart, and do not use its results to choose indicators, parameters, direction, or costs. Store the candidate hash and pass criteria before evaluation.

If a data correction changes holdout candles, record the provider revision and rerun as a new evidence event. Do not overwrite the original result. Reproducibility includes the data version, not only strategy code.

Walk-forward is a sequence of honest tests

An expanding or rolling walk-forward process repeats selection on past data and evaluation on the next untouched segment. Each segment must remain unseen until its candidate is frozen. Combining all folds after adapting to each test fold understates how much feedback influenced the rule.

Use equal-duration test windows when comparing fold metrics, track gaps, and distinguish a predeclared production-like protocol from exploratory analysis. A walk-forward report should show dispersion and failures, not only the aggregate curve.

Data leakage and the true trial budget

Leakage is broader than using tomorrow's close directly. Normalizing with full-period statistics, selecting symbols because their later history survived, revising labels after reading future bars, or allowing overlapping positions to cross the research/holdout boundary can all carry information forward. Document indicator warm-up, corporate actions or contract rolls, provider revisions, missing bars, and the timestamp at which each input would have been available. A chronological split is necessary but insufficient when feature construction already saw the future.

The trial budget must include human choices as well as automated parameter combinations. Looking at a chart and changing the date, removing a market, switching direction, or rewriting an entry phrase after each failure is still search. Keep a decision log with the reason and time of every adjustment. If several people or AI prompts explore candidates, count the combined opportunity to select a lucky winner rather than treating every session as independent.

Predeclare what follows pass, fail, and inconclusive outcomes. Pass advances the unchanged candidate only to observation; fail archives it without threshold repair; inconclusive waits for genuinely later data or a separately justified experiment. This policy removes pressure to rescue an attractive narrative after the holdout is open and makes the next action auditable before any performance number is known.

Use uncertainty language that matches the evidence. A profitable holdout with twelve trades is not validation at the same strength as several untouched windows containing varied regimes and a meaningful sample. Report raw counts, interval boundaries, rejected candidates, cost assumptions, and concentration rather than turning a binary gate into certainty. If the app or data provider cannot reconstruct an older run exactly, mark reproducibility as limited and do not present a later approximation as the original experiment.

Common mistakes

• Randomly shuffling time-series observations and allowing future information into training.
• Looking at the holdout chart while choosing rules or parameters.
• Trying many variants but reporting only the selected winner.
• Retuning after holdout failure and continuing to call the same window out of sample.
• Using a tiny holdout with too few independent trades to support a decision.
• Ignoring lookback overlap, open positions, data revisions, costs, or unrealistic fill assumptions.

Practical checklist

1. Write the hypothesis, trial budget, selection scorecard, and rejection conditions.
2. Assign chronological research, gap, and holdout boundaries before tuning.
3. Freeze data source, timeframe, costs, sizing, exits, and execution semantics.
4. Log every configuration and keep rejected outcomes.
5. Check parameter neighborhoods and realistic cost stress in research data.
6. Hash or snapshot the selected Backtest setup before opening the holdout.
7. Evaluate the holdout once and report pass, fail, or inconclusive without retuning.
8. Advance only the unchanged candidate to later closed-candle paper observation.

Related guides

Start with fair run comparison, inspect trade evidence without retrofitting, and use observation-only paper monitoring as a later test of the frozen candidate.

• How to Compare Backtest Runs Like for Like
• Trade Evidence: MFE, MAE, R-Multiple, and Holding Time
• From Backtest to Paper Monitoring on Closed Candles
• How Backtest calculates strategy tests

Research sources

• Lawrence Berkeley Lab: The Probability of Backtest Overfitting
• scikit-learn TimeSeriesSplit documentation
• TradingView Pine Script strategy documentation
• QuantConnect Backtest API reference