The Backtest Checklist: What to Verify Before You Trust a Result

A backtest is a simulation. It can be useful, misleading or completely fictional, and the difference is usually in the assumptions rather than the code.

This checklist is designed to be run before you trust a result — your own or a vendor’s. It covers the data, the design, the costs, the statistics and the out-of-sample test. It is the companion to AI for Trading Research and Overfitting: How AI Helps You Fool Yourself.

A backtest is a hypothesis test, not a track record. The output is evidence about the past, not a promise about the future.

1. The Hypothesis

[ ] Is there a stated reason the edge should exist? A behavioral, structural or liquidity explanation.
[ ] Was the hypothesis written down before the test?
[ ] Is the rule specified precisely enough for someone else to implement it?
[ ] Is the rule simple enough to be plausible, or does it have many parameters?

A pattern found by searching is a candidate, not an edge. A hypothesis with a mechanism is a better starting point.

2. The Data

[ ] Is the data source named and reproducible?
[ ] Is the sample period stated?
[ ] Is the frequency appropriate for the rule?
[ ] Are gaps, duplicates, bad ticks and timezone errors handled?
[ ] Is the data point-in-time? Does the universe include only instruments that existed at the time?
[ ] Are corporate actions, splits, dividends and contract rolls adjusted correctly?
[ ] Is the sample long enough to cover more than one regime?
[ ] Are overlapping trades or correlated instruments reducing the effective sample?

For futures, the roll method matters. For equities, delisted stocks and corporate actions matter. For crypto, exchange outages and stablecoin distortions matter. The data is the foundation; if it is wrong, the backtest is fiction.

3. The Design

[ ] Is there look-ahead bias? Does the rule use information that would not have been available at the decision time?
[ ] Is the entry at a realistic price, or at the close of the signal bar with no delay?
[ ] Is the exit defined: target, stop, time exit or trailing?
[ ] Is the position sizing defined: fixed, compounded or risk-based?
[ ] Is the universe defined: which instruments, sessions and liquidity filters?
[ ] Are the trade count and the average holding period reported?
[ ] Is the maximum drawdown reported, with its duration?
[ ] Is the worst month or worst week reported?

Look-ahead bias is the most common and most damaging error. A rule that “knows” the close before it happens can produce a beautiful, fake result.

4. The Costs

[ ] Are commissions included?
[ ] Is the spread included?
[ ] Is slippage included?
[ ] Are exchange fees included?
[ ] Is financing or swap included for overnight positions?
[ ] Are borrow costs included for short positions?
[ ] Does the edge survive at 2× the expected costs?
[ ] Does the edge survive at 3× the expected costs?

If the edge disappears at 1.5× costs, it is fragile. If it survives at 3× costs, it has a margin of safety.

5. The Statistics

[ ] Is the sample size sufficient for the frequency?
[ ] Is the result statistically significant after accounting for the number of tests?
[ ] Is the effect economically meaningful after costs?
[ ] Is the return distribution skewed or fat-tailed?
[ ] Is the drawdown acceptable relative to the return?
[ ] Is the Sharpe or Sortino ratio reported with its assumptions?
[ ] Is the result stable across sub-periods?
[ ] Does the result survive a change in the start or end date?

A high return with a 60% drawdown is not the same opportunity as a lower return with a 15% drawdown. The risk-adjusted result matters more than the headline.

6. The Overfitting Check

[ ] How many variations were tested? Count them.
[ ] Was the hypothesis pre-registered?
[ ] Is the rule simple relative to the sample size?
[ ] Do nearby parameter values work, or only the chosen one?
[ ] Does the result survive a walk-forward test?
[ ] Is there an untouched out-of-sample period?
[ ] Are the failures recorded, or only the winners?

The more variations you test, the more likely the best one is a spurious result. Bailey, Borwein, López de Prado and Zhu call this backtest overfitting and show how it inflates out-of-sample performance. The defense is pre-registration, simplicity, out-of-sample data and a research log that records the failures.

7. The Out-Of-Sample Test

[ ] Is there a reserved period the rule has never seen?
[ ] Was the final rule tested on it once, without further tuning?
[ ] Is there a walk-forward test with rolling windows?
[ ] Do the out-of-sample results degrade gracefully, or collapse?
[ ] Is the sample of out-of-sample trades large enough to be meaningful?

A single in-sample backtest is not enough. A walk-forward test is closer to live trading because it simulates the sequence of adapting to recent data and trading the next period.

8. The Execution Test

[ ] Has the rule been paper traded on live data?
[ ] Were the signals clear and unambiguous?
[ ] Was the execution reliable?
[ ] What was the difference between the signal price and the fill?
[ ] Were the costs close to the model?
[ ] Could the trader follow the rule through a losing sequence?

A backtest cannot test the trader’s psychology. Paper trading can. Move to capital when the process is stable, not when the curve is beautiful.

9. The Verdict

Write one of three outcomes:

Allocate: the hypothesis is plausible, the data is clean, the costs are realistic, the out-of-sample result holds and the drawdown is acceptable.
Iterate: the hypothesis is plausible but the rule or the data needs work. Specify the next test.
Abandon: the edge does not survive costs, out-of-sample data or robustness checks. Record why.

The verdict should be written down with the date and the evidence. A research log that records only the winners is a catalogue of overfitting.

A Worked Example

A trader tests a mean-reversion rule on MNQ.

Hypothesis: short-term reversal after a liquidity sweep.
Rule: enter at the next open after a sweep of the prior session low and a close back inside the range; exit at the prior session midpoint or the stop.
Data: five years of 5-minute bars, with contract rolls handled explicitly.
Costs: commission, one tick of slippage and the spread.
Result: 1,200 trades, profit factor 1.3, maximum drawdown 18%, average R 0.12.

The checklist flags:

The rule was tested on 40 parameter variations; the best result is the maximum of 40 draws.
The out-of-sample period was not reserved.
The result at 2× costs is a profit factor of 1.05, which is fragile.
The drawdown is 18%, which may be acceptable, but the average R is small.

The verdict is iterate, not allocate. The next test reserves two years of out-of-sample data and fixes the parameters before running.

Common Mistakes

Testing without a hypothesis.
Ignoring costs.
Using look-ahead data.
Optimising on the full sample.
Testing too many variations and believing the best.
Ignoring drawdown.
Confusing a backtest with a track record.
Trusting a vendor’s equity curve without the data and the method.
Skipping paper trading.
Not recording the failures.

Frequently Asked Questions

What Is A Backtest?

A simulation of a trading rule over historical data. It is a hypothesis test, not evidence of future returns.

How Much Data Do I Need?

There is no universal number. It depends on the frequency, the instrument and the effect size. More data does not fix a biased sample.

What Is Look-Ahead Bias?

Using information in the test that would not have been available at the decision time. It is the most common cause of a beautiful, fake backtest.

What Is Overfitting?

Fitting a rule to the noise in a sample, producing a result that does not generalize. It is more likely with many variations, small samples and complex rules.

What Is Walk-Forward Testing?

Optimising on a rolling training window and testing on the next window, then rolling forward. It is closer to live trading than a single in-sample backtest.

Is A Backtest Enough Before Going Live?

No. Paper trade first, then start with small size. A backtest cannot test the data feed, the execution or the trader’s psychology.

Can AI Validate A Backtest?

It can check the design and flag common errors, but it cannot certify the data or the result. Verify the data and the assumptions independently.

Sources

Bailey, D. H., Borwein, J. M., López de Prado, M., and Zhu, Q. J. (2014), “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance,” Notices of the American Mathematical Society, 61(5), 458–471.
Bailey, D. H., Borwein, J. M., López de Prado, M., and Zhu, Q. J. (2017), “The Probability of Backtest Overfitting,” Journal of Computational Finance, 20(4), 39–69.
Arnott, R., Harvey, C. R., and Markowitz, H. (2019), “A Backtesting Protocol in the Era of Machine Learning,” Journal of Financial Data Science, 1(1), 64–74.
US Commodity Futures Trading Commission, Customer Advisory: AI Won’t Turn Trading Bots into Money Machines (25 January 2024). AI cannot predict the future; guaranteed returns are a fraud red flag.
CME Group, Micro E-mini Nasdaq-100 (MNQ) contract specifications, used for the synthetic example only. Verify current specifications with the exchange.

No statistic in this article is invented. Verify figures against the primary source before republication.

Trading involves substantial risk of loss. This article is educational information, not financial, investment, legal or tax advice, and it is not a personal recommendation or solicitation. The example is synthetic and illustrative. Backtests and AI-generated research are not guarantees of future results. Verify every data source, assumption and result independently.