Your Backtest Is Lying to You
I built a scoring and backtesting system for my own investing: a multi-factor stock ranker on one side, a replay engine on the other. The first equity curve it produced was gorgeous. Smooth, steep, barely a drawdown. I did not feel clever. I felt suspicious, because I have written enough systems to know that when a program tells you something wonderful on the first run, it tends to be describing its own defects rather than the world.
It was. The curve was a bug report written in the notation of finance. When I dug through it, most of the failures turned out to be systems problems I had already met in other codebases, with a ticker symbol attached.
Look-ahead bias, the version that gets you
You will never write buy(tomorrow_close) on purpose. The leaks that make
it into production are quieter:
- Indicators computed on the full series, then sliced. You load ten years of prices, compute a moving average or a z-score over the whole array, and then iterate day by day. Every value in that array carries knowledge of the entire history, including the part that had not happened yet at the simulated date, so the loop looks causal while the data is not.
- Deciding during a bar on that bar's close. A rule that says "if today closes above the average, buy today at the close" only executes if you knew the close before it printed. Signal time and execution time run on different clocks, and honest backtests lose most of their edge in the gap between them.
- Fundamentals applied before publication. A P/E ratio dated to the end of the fiscal quarter is a ratio no investor could compute until the filing came out, weeks later. Ranking a universe on data that was still confidential amounts to time travel.
- Restated financials. Vendors serve you the corrected numbers by default, so your backtest trades on figures that someone revised after the fact, in the direction reality later confirmed.
I stopped relying on vigilance, because vigilance fails at 1am when you are adding one more feature. The engine needs an invariant it cannot violate: at simulated time T, the only data reachable is data whose timestamp is less than or equal to T, enforced by the API rather than by the author's discipline. That means point-in-time storage with two timestamps per record, the event time and the time it became knowable, plus an accessor that takes T and has no way to return anything newer. Once you make the wrong access unrepresentable, that class of bug stops recurring instead of moving around.
Survivorship: your universe is a list of winners
Ask where your universe of stocks came from. If the answer is "today's index members" or "everything my data provider currently lists", the backtest is running on companies pre-selected for the property of still existing. Your provider has already dropped from history everything that went bankrupt, got delisted, or was acquired at a discount, and those were the positions that would have hurt.
Testing "the S&P 500 constituents" with today's membership list compounds the problem, because index membership itself is a momentum and quality filter. Companies join after doing well and leave after doing badly. Backtesting today's members over the last decade means buying, in 2016, the firms that a committee would later decide had earned their place. You need historical constituent lists with the dates they changed, and a universe that includes the dead. If the strategy only works on survivors, you have measured survival rather than skill.
The point-in-time problem, generally
Survivorship and look-ahead share a root cause: the historical record you query is not a fixed object. Vendors keep rewriting it under you.
- Dividend and split adjustments retroactively change every prior price in the series, so the number your rule compared against last month is not the number it compares against today.
- Vendors revise without announcement. A backfill, a corrected corporate action, a changed methodology, and yesterday's file no longer matches today's file.
- Tickers get reused. The same three letters can be two unrelated companies across a decade, which is why identity should hang off a stable security identifier and never off the symbol.
Overfitting: the best of 200 tries is a measurement of noise
Search a parameter grid of 200 combinations and report the winner, and you have run a lottery and reported the ticket that hit, whatever you call the result. The more configurations you test, the higher the best in-sample result you should expect from pure chance, which means the winner's performance needs to be discounted by how hard you searched before it means anything. Two hundred coin-flippers, one of them lands eight heads in a row, and there is nothing special about his wrist.
The countermeasures are unglamorous. Hold out data and refuse to look at it until the end. Use walk-forward: fit on a window, trade the next window blind, roll, and read only the stitched out-of-sample result. Count every experiment, including the ones you abandoned, because the abandoned runs are part of the search. And remember that "my strategy beat buy-and-hold from 2015 to 2020" is a single sample from a single regime, reported by someone who chose the dates after seeing the data.
Costs the naive engine forgets
The first version of any engine fills every order instantly at the close, in unlimited size, for free. Reality charges for each of those assumptions:
- Spread: you buy at the ask and sell at the bid, so you pay the spread on every round trip.
- Slippage and market impact: your own order moves the price against you, more so as size grows and liquidity thins.
- Borrow costs: short legs cost money to hold, and the hardest-to-borrow names are the ones the signal loves most.
- Taxes and commissions: jurisdiction-specific, turnover-sensitive, and brutal to high-frequency rebalancing.
A strategy whose per-trade edge is smaller than the spread loses money on every round trip by construction. And an engine that fills you at prices no one was offering has stopped simulating the market you plan to trade in.
Regime dependence: n equals one
Markets are not stationary. The relationships your factors exploit are conditional on a monetary and structural backdrop that changes on a timescale of years. A ten-year backtest feels like a lot of data because it contains thousands of daily bars, but on the variable that dominates asset returns, the interest rate cycle, it may contain exactly one observation. You get thousands of samples of the noise and one sample of the variable that drives returns. Ask what happens to your strategy across a regime it has never seen, and if the honest answer is "unknown", write that in the report.
The engineering frame
One reframing changed how I built mine: a backtest is a replay engine. It belongs to the same category of system as an event-sourced service that must reconstruct the past faithfully, and the properties that make those services trustworthy carry over one for one:
- Deterministic replay: same inputs, same seed, same outputs, on every run. If two runs of the same configuration differ, you are not taking measurements.
- An immutable event log: market data and fundamentals arrive as appended facts, never as updated rows, and corrections come in as new events with their own knowable-at timestamp.
- No mutation of history: the engine computes adjustments at read time from the event log instead of baking them destructively into the stored series.
- Clock discipline: event time and processing time live in different fields with different meanings, and confusing them is the whole of look-ahead bias restated in distributed-systems vocabulary.
- Pinned, versioned datasets: every run records the snapshot identifier, the code commit and the parameters, so you can reproduce a result months later or declare it void.
If you cannot rerun a result byte for byte from a pinned dataset, treat the number as a property of your pipeline rather than of your strategy.
What I hold my own engine to
Point-in-time access comes first: at time T the engine can reach nothing timestamped after T, and the API enforces that rather than my care. Signal time and execution time stay separate, with a realistic gap between them. The universe includes delisted, bankrupt and acquired names, reconstituted from historical membership lists with dates, and identity keys off a stable security identifier, never off the ticker.
I model every cost the naive engine skipped, spread, slippage, impact, borrow, commissions and taxes, and then ask whether the edge survives them. I quote out-of-sample and walk-forward results and nothing else, and I count each experiment I ran, abandoned ones included. Each run pins a dataset snapshot plus a code commit, and a rerun has to match byte for byte. The report states which regimes the test covered, and which it did not, which is the more useful line of the two.
These constraints leave a backtest short of true, but they make it falsifiable, and falsifiable is the most a simulation of the past can offer. When the next gorgeous curve appears, I treat it as the null hypothesis and spend a week trying to break it, and I am pleased when I succeed, because a leak found in my own engine costs me a week while the same leak found in the market costs me money.