Random signals can look great
Each grey line is ten years of a strategy with zero true edge: random scores, real-looking volatility.
The orange one is simply the best of 25. Shown alone, it would pass for a decent strategy.
The null
What luck looks like
Simulate strategies that have no edge by construction. Their spread is the yardstick for yours.
Run a thousand of them
Simulate 1,000 no-edge strategies and plot their Sharpe ratios. This is the null distribution: what ten years of pure luck produces.
Most land near zero, but the tails reach further than intuition suggests.
Place the real signal
Residual momentum's Sharpe of 0.9 sits far in the right tail. The share of null runs at or above it is the p-value.
Judged as a single test, it's very unlikely to be luck.
But you didn't run a single test
The best of those 1,000 random strategies is marked too. It's close to the real signal, and can beat it.
This is page 4's multiple-testing problem, seen directly. Searching enough noise finds something as good as your discovery.
Rule of thumbCompare your result to the best of as many null runs as variants you tried, not to the average null run.
25 strategies with zero true edge · cumulative return, %
True edge
0
every path
Best path's Sharpe
0
Paths ending up
0
of 25
1,000 no-edge strategies · 10-year Sharpe ratio
Null std dev
0
≈ 1 ÷ √years
Real Sharpe
0.90
p-value
0
share at or above 0.90
Best null run
0
The bootstrap
How sure is the Sharpe?
A Sharpe ratio is an estimate from one sample. Resampling that sample shows how far the true value could be from it.
Resample the months
Take the 120 monthly returns and draw 120 of them at random, with replacement. Compute the Sharpe. Repeat 2,000 times.
Each draw is a history that's just as consistent with the data as the one you saw.
Read the interval
The middle 95% of those resampled Sharpes is the confidence interval. Here, ten years of a 0.9 Sharpe gives about 0.25 to 1.65.
The interval excludes zero, but it's wide. "Sharpe 0.9" really means "probably somewhere between a quarter and one and a half."
Short track records say little
The interval shrinks with the square root of time. Slide the track record down to three years and zero is back inside it.
Uses Lo's standard error, √((1 + SR²/2) ÷ years), with an observed Sharpe of 0.9.
When months aren't independent
A plain bootstrap assumes each month is a fresh draw. Momentum returns cluster in regimes and crashes, so that assumption understates the uncertainty.
Interview checkUse a block bootstrap: resample runs of consecutive months, such as 6 or 12 at a time, so the clustering survives. The interval gets wider, and more honest.
2,000 bootstrap resamples · Sharpe ratio
Observed
0.90
95% interval
—
Resamples below 0
0%
95% interval for a 0.9 Sharpe · by track record length
Standard error
0
Interval
—
t-stat
0
The cone
What a good signal's bad year looks like
Simulate many futures for a signal whose edge is real. The spread tells you which drawdowns are normal and which mean it's broken.
One path
Here is five years of a strategy with a true Sharpe of 1.0 and 10% annual volatility. Its edge is real by construction.
It still has stretches that look like failure.
Five hundred paths
Simulate 500 futures with the same edge. The shaded bands hold the middle 50% and 90% of outcomes at each month.
Even a real Sharpe of 1.0 can be flat after two years. Losing roughly one calendar year in six is normal.
Change the edge
Drag the true Sharpe and watch the cone and the drawdown statistics move.
Annual volatility is fixed at 10%.
Read a live drawdown
The orange line is a live track record after launch. It's down for most of its first year and alarming on its own.
Against the cone it sits inside the 90% band, so it's consistent with the backtest. Turning it off here would be selling at the bottom of a normal drawdown.
Rule of thumbWrite the kill rule before launch, from the cone. For example: review if the live path spends three months below the 5th percentile. Then follow it.
Cumulative return, % · 60 months
Down after 1 year
0%
of paths
Any losing year
0%
within 5 years
Median max drawdown
0%
Worst 5% drawdown
0%
or deeper
The calibration
Test the machine, not just the signal
Build fake data where you know the answer, run your real pipeline on it, and check that it finds exactly what you planted.
Plant a known signal
Simulate 300 stocks for 60 months where scores predict returns with a planted IC of 0.03. Run the exact code you use on real data.
Each dot is one run. If the pipeline is sound, the dots scatter around the diagonal: what you recover matches what you planted.
Plant nothing
Now plant an IC of zero. A sound pipeline should find nothing, and the dots should center on zero.
This is the false-positive check. It catches more bugs than any amount of staring at code.
Introduce a bug
Switch on a common bug: the score accidentally includes part of the month it's predicting, as when the skip month is dropped.
With nothing planted, the pipeline "finds" an IC far above anything real. Page 3's rule of thumb, that an IC of 0.20 usually means a bug, is this.
Try it yourself
Change the planted IC and toggle the bug. Each change runs fresh simulations in the page.
Say it this way"Before trusting the backtest, I ran the pipeline on synthetic data. It recovers a planted IC of 0.03 within noise and finds nothing in pure noise."
Pipeline
30 runs · planted IC vs recovered IC
Planted
0.030
Recovered, mean
0
Run-to-run spread
0
std dev