Signal research for alpha · Page 4 of 6
A signal that survived page 3 has earned an attack, not a launch. Check the data for lies, count how hard you searched, test on data you never touched, and see whether it holds up everywhere the story says it should.
The data
Before any statistics, check that the backtest only used what you could have known, about stocks you could have owned.
A number has the date it describes, the date it was published, and the date it reached your system. A quarter ends in March, the company reports in May, the vendor loads it the next morning.
A backtest may only trade after the last of the three. Using any earlier date is look-ahead.
The most common version is quieter. A signal built from today's closing price can't also trade at today's close, because the price wasn't final until the market shut.
For fast signals, moving execution one day later can cut the Sharpe by more than half. Residual momentum skips a month, so it's mostly protected. Still check.
Build a universe from today's stock list and you've quietly dropped every company that went bankrupt or got delisted.
In this simulated universe of 300 stocks, about a quarter disappear over ten years. The survivors average several points a year more than the stocks you could actually have bought.
For a long-only strategy, survivorship flatters returns. For momentum, the missing stocks are often losers that would have been profitable shorts, so the bias can also hide performance.
Either way, the backtest describes a market that never existed.
The search
You didn't run one backtest. You ran every variant you looked at, and the best of many lucky draws looks like skill.
Lookback of 6, 9 or 12 months. Skip a month or not. Regression window of 24, 36 or 60 months. Three neutralizations, three universes, two weightings.
Each choice multiplies the others. With none explored, t = 2.3 passes. Tap the choices you'd really have tried and watch the hurdle rise.
Run that many variants on pure noise and the best one still shows a respectable t-stat. The orange curve shows how high it typically gets.
With all 324 variants, noise alone typically produces a best t of about 3.1.
A Bonferroni-style correction lifts the significance hurdle with every trial. Harvey, Liu and Zhu argue a new factor should clear t > 3 for this reason.
Residual momentum's IC t-stat of 6.7 clears any bar. Its alpha over plain momentum, t = 2.3, fails the corrected hurdle if you really searched 324 variants.
The only defense is honesty about the count. Log every variant you run, including the ones you abandoned, and report the number with the result.
The holdout
In-sample performance shows how well you fit. Out-of-sample performance is the only evidence of how well you'll do.
Lock away the last few years before you start. Do all research on the rest, then test on the holdout exactly once.
Look at it twice and it becomes training data. One holdout, one shot.
Fit on everything up to a date, test on the next year, roll forward, repeat. Every test year uses only the past.
This is how the strategy would really have run, refit as data arrived.
Standard cross-validation trains on data from both before and after the test fold. Financial data is serially dependent, so the model learns from the future around the test period.
Residual momentum's 12-month formation window makes it worse: training rows just after the test fold are built from test-fold returns.
Purge training rows whose windows overlap the test fold. Add an embargo, a gap after the fold, so serial correlation can't carry information across.
López de Prado's purged cross-validation makes k-fold usable for overlapping financial labels.
The stress test
A real effect shows up in most slices of the data, and most strongly where its story says it should.
Break the sample into periods. A real signal is positive in most of them, even if it's weaker in some.
Here one stretch loses money: a momentum crash, when last year's losers rebound sharply. The other periods carry it.
Page 2 predicted that underreaction should be strongest where news travels slowly. Split by size and by analyst coverage.
The IC is highest in small, thinly covered stocks, which is what the story said. A prediction that comes true is worth more than another decimal of t-stat.
Vary the lookback and the regression window around your choice. A real effect gives a plateau: neighboring settings work almost as well.
An overfit signal works at one exact setting and nowhere near it. That isn't a discovery about markets. It's a discovery about one corner of your sample.
Residual momentum survives the attack on most counts. The open question is luck, and page 5 answers it with simulation.