It Survived 5 Years in Backtest, Then Blew Up in 3 Months: The Survivorship Trap

The illusion of the proven backtest

With 358 indicators and 741 assets, you can always find a combination that looks spectacular on past data. That's not evidence it works — it's selection. Search enough variants and some will fit history by chance, then fail the moment they meet data they weren't fit to.

Our own numbers show the gap: only 26% of indicator combos beat buy-and-hold once we tested them out-of-sample and included costs. The other ~74% looked fine until held to that standard.

Why 'it worked for 5 years' fools you

A backtest over one long period is a single sample. If you tuned the rules to that period (even unconsciously, by picking what 'worked'), the 5-year result is hindsight, not a forecast. Markets also change regime — a trend-following rule that thrived in a bull run can bleed out the moment conditions shift. The backtest can't show you the regime it never saw.

How to not get fooled

• Out-of-sample only. Trust a result only on data the strategy was never tuned on.
• Costs included. Frictionless backtests are fantasy.
• Beware the search. The more variants you tried, the more likely the winner is luck — penalize it accordingly.
• Expect regime change. Size and risk-manage as if the good period will end, because it will.

Every figure on this site is built this way on purpose — it's why our numbers look humbler than a course's, and why they're more likely to survive contact with a live account.