Overfitting backtest: Two people on Reddit pushed back on my last post about testing 324 parameter combinations. Both were right, and their pushback turned into the most useful test I’ve run on this bot so far.
Here’s what they said, what I did about it, and what the corrected numbers actually show including the uncomfortable parts.
The critique
After I posted that the best of 324 backtested combinations returned +68.6%, someone pointed out the obvious flaw I’d skipped: testing 324 combinations against the same data and reporting the best one is a textbook multiple-comparisons problem. With enough combinations, something will look great by chance alone, whether or not there’s real signal underneath.
A second commenter, who works on market data professionally, laid out exactly what a proper check would look like:
- Look at the distribution, not the max. If most results cluster in one range and a handful spike, the spike is probably partly noise.
- Check whether a “working” parameter shows a smooth trend or an isolated peak. A gradual, explainable pattern is more trustworthy than a lone winning combination.
- Validate out-of-sample. Select parameters on one slice of data, then confirm the result holds on a separate slice the selection process never saw.
I hadn’t done any of this. So I did all three.
Running a Proper Overfitting Backtest: What Changed
Running the same 324 backtests and keeping every result instead of just the best one:
- Median return: -8.7%
- Maximum return: +62.6% (the number I’d originally reported)
- Gap between median and max: 71 percentage points
Most parameter combinations lost money. The headline number from my last post was near the extreme edge of the distribution, not a representative outcome. This alone was enough to know the original framing was misleading, even before running the other two checks.
The ADX finding didn’t survive scrutiny
In my last post, I argued the ADX threshold effect was trustworthy because it looked directionally consistent lower ADX outperforming higher ADX across many combinations, which I read as a smooth, explainable pattern rather than a lucky spike.
Checking it properly: ADX 20 averaged -5.4%, ADX 25 averaged -4.2%, ADX 30 averaged -6.2%. That’s not a monotonic trend. ADX 25 the setting I’d previously moved away from was actually the best performer of the three in this stricter test. What I’d presented as a reliable, explainable signal did not hold up once I looked at it correctly.
The out-of-sample test is where it really broke down
I split the data: the first two-thirds for parameter selection, the last third held back entirely, never used for choosing anything.
The “best” in-sample combination:
- In-sample: +62.6%
- Out-of-sample: -8.9%
- A 71-point collapse. This is what overfitting looks like in practice a combination that performed well because it happened to fit the noise in that specific data, not because it captured anything durable.
My current live bot’s actual settings (which weren’t the single best in-sample cell, but came from separate, earlier reasoning):
- In-sample: +50.2%
- Out-of-sample: +16.3%
- Still a real decline, but it held a positive edge on data it had never seen.
What I’m taking from this
The uncomfortable part first: the headline number from my previous post was not a reliable estimate of anything. It was close to the best-case noise in a 324-way search, and I reported it as if it were a finding. That’s a mistake worth naming plainly rather than softening.
The more useful part: my actual live trading bot which wasn’t selected by picking the top cell of that grid search, but from a smaller, earlier round of reasoning about the strategy held up meaningfully better under the same out-of-sample test than the “statistically optimal” combination did. That’s not proof the current settings are correct going forward, but it shows why properly checking for overfitting backtest results matters before trusting a number.
Going forward, any parameter change to the live bot needs to clear the same bar: does it hold on data it wasn’t selected on. A backtest number without that check is not evidence, it’s a claim.
Thanks to the people who pushed back on the last post instead of just upvoting it. That’s the useful version of internet feedback, and it’s the reason this post exists.
This is the third post on AI Project Log. about overfitting backtest The first post covers the build; the second covers the original parameter search this one corrects.
