How To Tell If Your Backtest Is Luck

I designed 62 intraday trading strategies, tested every one on both halves of a train/test split, and collected the 116 resulting t-statistics.

Mean: +0.060. Standard deviation: 0.96.

Pure random noise produces exactly 0.00 and 1.00.

I did not build a portfolio of weak edges. I built a random number generator out of 62 ideas that each had a plausible story about why they should work. That distribution is the most useful thing I own, because it tells me what my own failures look like — and therefore what a real result has to beat.

Not one of the 116 exceeded +3.

Why t = 2 is not the bar

The number most retail backtesters treat as "it works" is t = 2. Roughly a 5% chance of arising by luck, if you ran one test.

Nobody runs one test.

I ran 22 strategies in a single round. With 22 independent comparisons there's about a 68% chance that at least one clears t = 2 purely by chance. Not a small risk — more likely than not. If I'd run 22 worthless ideas and published the best one, I'd have had a two-in-three shot at a publishable-looking result built on nothing.

So the bar has to move with the number of attempts. For 22 tests it's roughly t > 3.04. For the 40-strategy round it's about t > 3.23.

This is the correction almost nobody applies, and it's why "my backtest scored 2.4" is not information. The score is uninterpretable without the denominator. How many did you try before this one? Including the ones you abandoned after ten minutes because they looked bad? Those count too.

Of my 40-strategy round, the best result measured by its weaker half was +1.17. Nothing came close.

Three ways I nearly fooled myself

Half a result. Three separate times a strategy failed one half and scored well on the other:

strategy	train	test
rotate inside value	−0.21	+2.54
VWAP reversion	−0.22	+1.84
VWAP band ride	+1.74	+0.65

Each of these is tempting, and each has an available excuse — regime change, the market evolved, the early data is stale. The excuse is sometimes even true. But if I accept it here I have to accept it everywhere, and then I have no test at all. Both halves or nothing.

Post-hoc sign flipping. One strategy — volume dry-up followed by a break — was consistently negative: −2.01 and −2.53. Reversing it gives +2.38 / +1.77. Free edge, surely?

No. It's short of the 3.23 bar, and more importantly it's post-hoc: I only thought to flip it because I saw it lose. That's the same as picking the winner out of the pile, one step removed. Worth noting it isn't even a clean sign flip mechanically — with a 1.5R target against a 1R stop, the payoff geometry isn't symmetric, so "reverse the signal" doesn't simply negate the result.

A great ratio on almost no events. A 200-day trend filter posted a Sharpe of 1.25 — better than anything else I tested — off a skill score of +0.30 on five trades. It scored well by being out of the market, not by choosing anything. Ratios don't tell you how many decisions produced them.

The bug that looked like a finding

My first run of the 22-strategy round reported 11 of them as "too few trades." I briefly took that as a result: these ideas just don't trigger often.

It was a bug. ATR was being computed within the current session only, so it returned nothing before bar 14, and every strategy that entered before 10:40 was silently dropped — which happened to be most of the opening-based ideas, the ones I most wanted to test.

A backtest that returns no trades is almost always a bug, not a finding. More generally: when a result is strange, the probability that your harness is broken is much higher than the probability that you discovered something about markets.

What actually survived

Out of everything, one intraday strategy scored +4.06 — outside the entire measured distribution of my 116 failures.

That's the comparison that convinced me, and it's stronger than the raw score. It didn't just clear a threshold. It cleared a threshold that 116 deliberate, mechanism-backed attempts by the same person, on the same data, with the same harness, never came close to.

If you only ever test the thing you believe in, you have no idea what your own noise looks like. Test forty things you don't believe in. The distribution you get back is the yardstick for everything you do afterward.

The limit I have to state

The held-out half has now been queried 62 times. It is no longer a clean test set. Every additional strategy I score against it degrades it further, and any future winner should be discounted accordingly — by me first.

Real validation from here needs data I haven't touched: a different instrument, or forward time. That's the honest cost of a large search, and it's why I stopped searching rather than running a fifth round.