Data snooping is testing idea after idea on the same chart until one looks great. Try enough versions and one will win, even when none of them has an edge. The winner looks real because it's the only one you look at.
What is data snooping?
Testing lots of versions on the same data and keeping the best one. Think of every test as a lottery ticket. Buy enough tickets and one of them pays, by luck alone, and that's not what you want, right? Overfitting is a bit different. It modifies one strategy until it fits the noise and makes money. Data snooping picks the luckiest of many. Both end in the same place, a backtest that doesn't work live.
How does it happen?
Usually without you noticing. Each step feels reasonable.
| What you do | Why it feels fine | What it really does |
|---|---|---|
| Try 50 settings and keep the best | each test only takes a minute | the best of 50 is mostly luck |
| Add filters until the curve is smooth | every filter has a reason | the reasons came after the results |
| Test ten ideas and share the winner | you only show what worked | the nine that failed still count |
| Reuse the same years for every new idea | it's the data you have | the data stops being a fair test |
| Let AI tools test hundreds of versions | it feels professional | it finds lucky winners faster |
How do you catch it?
Count every version you tried, including the ones you're not using now. The more you tried, the less the winner means. Keep a part of the data you never touch until the very end, that's your out of sample test. Then check it again with walk forward. A monte carlo shuffle shows how often luck alone gets the same result. If the edge only shows up in the one version you picked... yeah, it's probably luck.
Why it's the hardest one to see
It isn't in the data and it isn't in the code. It's in your process. Nobody sees the 49 versions you threw away, sometimes not even you. That's why it sits next to survivorship bias in the list of things that make a backtest look better than it is.
Frequently asked questions
Related terms
- Expectancy — What one average trade is worth to you, after the wins and losses cancel out.
- Overfitting — When a strategy is tuned so tightly to the past that it only works on the past.
- Out of Sample — The data your strategy never saw while you built it, kept back to test if the edge is real.
- Walk Forward — Out of sample testing done over and over, rolling through your whole history instead of one split.