The Backtest That Never Happened: How Historical Simulations Learn What No One Could Have Known

Every backtest is a story about a past that never quite happened. The simulation buys and sells in a market reconstructed from data files, and the reconstruction differs from the market that existed in ways that are easy to miss and consistently flattering. Securities that later failed are absent from the universe. Economic figures appear in their final, revised form rather than the form first printed. A quarterly filing is stamped with the quarter it describes rather than the day it was released. None of these is a bug in the strategy. Each is a way the historical record quietly tells the simulation something that no one trading at the time could have known.

This piece catalogs the main channels through which the future leaks into a historical simulation, shows in stylized form how large the resulting distortions can be, and describes the point-in-time data discipline we consider a precondition for taking any backtest seriously. The theme throughout is that a simulation is only as honest as the answer to a single question, asked of every number it consumes: when, exactly, did this become knowable?

The Universe That Forgot Its Losers

The oldest and best-documented leak is survivorship. A researcher who builds a universe from the securities that exist today, and then runs a strategy backward through time on that list, has selected on the outcome. Firms that went bankrupt, were delisted for failing listing standards, or were acquired at a discount have been removed; the ones that remain are, by construction, the ones that made it. A long-only strategy tested on such a universe looks better than it was. A long-short one is distorted less obviously: its long leg is flattered for the same reason, while its short leg is penalized, because the names it would have shorted to zero are among the ones that are gone. The net effect depends on the strategy, which is exactly why it cannot be corrected after the fact.^[1]

The leak has a close cousin. Even when a database retains a security until its delisting, it often lacks the final return: the price on the last day of trading is recorded, but what a holder actually received afterward, frequently a small fraction of that price, is not. A simulation that holds the position at its last quoted price, or silently drops it at no cost, has assumed away exactly the loss a real portfolio would have taken. A classic study of the problem found that supplying the missing delisting returns changed the conclusions of a well-known body of research on small firms.^[1]

Figure 1 stylizes both effects together. The same rules are run on two universes. The first is the universe as it existed each month, with removed securities held until their delisting and their delisting return charged. The second is the survivors-only universe common in casual research. In the stylized model the survivor universe overstates the monthly return by a constant drag that the removed names would have contributed, and the drag is larger in a stress window, because delistings cluster in downturns. The two paths part company slowly at first and then decisively.

Two features of the picture generalize. The overstatement compounds, so its size in a long backtest is much larger than the modest monthly figure suggests. And the survivor universe understates the drawdown as well as inflating the level, which matters more, since risk limits, capital allocation, and the decision to keep trading through a bad period are all set from the drawdowns a backtest shows.

The Number That Changed After You Read It

Survivorship is a leak about which securities existed. A second family of leaks concerns what was known about them, and its cleanest example is macroeconomic data. Output, employment, inflation, and most other aggregates are published as a first estimate and then revised, sometimes for years, as late reports arrive and seasonal factors are recomputed. A database that stores one value per period stores the latest one, and a simulation that reads it is trading on a figure that did not exist on the date the simulation thinks it did.^[2]

The revisions are not noise around the first print. They are systematically informative at exactly the moments that matter. Around turning points, first estimates tend to lag: a downturn appears milder in the initial release than in the revised record, and the recovery appears later. A strategy that conditions on revised data therefore reacts to a recession before a real-time investor could have seen it, and its apparent skill at calling turns is partly the skill of the statistical agency's later corrections.

Figure 2 shows a stylized version. The underlying series is a smooth cycle with one sharp downturn. The first print is a blend of the current true value and the value two quarters earlier, plus a little noise, so it lags the truth; the revised series is the truth plus smaller noise. The gap between the two lines is what a backtest run on revised data knows that no one knew at the time.

The same mechanism operates on company fundamentals, where it is if anything more severe. Restated financials replace the originals in most databases, so a simulation that ranks companies on earnings quality may be ranking on numbers corrected after the fraud or the accounting change came to light.^[3] The remedy is the same in both cases: store every vintage of every figure, and let the simulation see only the vintage that existed on its clock.

When Was It Known?

The revision problem is a special case of a more general one: the timestamp attached to a piece of data usually describes the period the data refers to, not the moment it became available. A fiscal quarter ending in March is reported in April or May. An index reconstitution is announced days before it takes effect. An analyst forecast dated to a Monday may have entered the vendor's system on Wednesday. When the simulation joins any of these to prices by the nominal date, it trades on the information before the market received it.

Corporate actions and identifiers leak in subtler ways. Splits and dividends require price adjustments that must be applied from the ex-date forward; a series adjusted backward from today is fine for a chart and wrong for a signal, because the adjustment factor itself encodes future events. Ticker symbols are reused, so a strategy that keys on symbols can splice the history of a defunct company onto its successor. And a universe defined by present index membership commits survivorship in another guise, since the constituents of an index today are the companies that grew into it.

Figure 3 collects the channels we watch for. The list is not exhaustive, and no single item on it is exotic. What they share is that each is invisible in the finished backtest, which reports only that the strategy worked.

An Audit, in Stylized Numbers

How much do these leaks matter? The honest answer is that it depends on the strategy: a signal built from prices alone is largely immune to revision bias and highly exposed to survivorship, while a macro strategy is closer to the reverse. It is nonetheless useful to see what an audit of a single leaky backtest can look like, and Figure 4 presents one in stylized form. Each leak is assigned an illustrative overstatement of the strategy's annualized return, measured two ways: the reduction when that leak alone is repaired, and the further reduction when it is repaired last, after all the others.

Two lessons come out of an exercise like this, and both hold beyond the invented numbers. First, the leaks overlap. Survivorship and missing delisting returns remove much of the same performance, so fixing them in sequence yields less than fixing either in isolation, and the sum of the standalone bars overstates the total. Second, the largest items are rarely the famous ones. Timestamp contamination in ordinary fundamental and event data is easy to introduce, hard to see, and in our experience the leak most likely to explain a backtest that looks too good.

There is also a compounding interaction with selection. A researcher who tries many variants and keeps the best has already tilted the result upward before any leak is added; a leak that inflates every variant by a little inflates the chosen one by more, because the selection favors the variants in which the leak happened to help most.^[4] Data discipline and multiple-testing discipline are not separate hygiene practices. Each makes the other's failures worse.

A Pipeline That Cannot See Forward

The remedy for all of this is not vigilance. Vigilance fails quietly, one join at a time. The remedy is an architecture in which the simulation cannot obtain a number that was unknowable on its clock, because no code path exists that would return one. Figure 5 sketches the pipeline we consider the minimum.

Its principles are few. Raw data is kept as delivered and never overwritten, so that every vintage remains available. Every record carries two dates, the period it describes and the moment it became knowable, with the second set conservatively when it is uncertain. Revisions accumulate in a table keyed by both dates rather than replacing earlier values. Identifiers are mapped through a history that never reuses a symbol. Corporate actions are applied from the ex-date forward. Universe membership is reconstructed as it stood on each date, including the securities that were later removed and their terminal returns. And the simulation reaches the data through exactly one operation, an as-of join that returns the latest vintage whose knowable date precedes the simulation clock, and through nothing else.

A pipeline built this way is slower to assemble and more expensive to run than a flat table of latest values, and the backtests it produces are, almost without exception, less impressive. That is the point. A strategy that survives point-in-time simulation has been tested against the past as it was actually experienced, which is the only past the future will resemble. Every other backtest is a backtest that never happened.