I Ran Four Different Statistical Tests on My Trading Bot. They Didn’t All Agree.
A week ago I posted here about walk-forward validation after this community rightly called out that a single out-of-sample split isn’t enough to trust a backtest. Since then, three more people pushed on three more angles, and I tested all of them. Here’s the full picture, including the parts that don’t fit together as neatly as I’d like.
Where this started
Quick recap for anyone new here: I built a crypto trading bot, ran a 324-combination parameter search, reported the best result (+68.6%) with zero validation, and got rightly torn apart for it. Since then I’ve been rebuilding the validation process from scratch, one piece of feedback at a time.
Test 1: Walk-forward across independent windows
Instead of one in-sample/out-of-sample split, I cut three years of data into six sequential windows and tested the same fixed live settings on each one, independently, without re-optimizing between them.
Result: 5 of 6 windows positive, median +10-12% per window. The one negative window had only 2 trades in it — not enough signal to judge either way.
This answered “is the edge consistent over time.” It didn’t answer whether the edge itself was statistically real, or whether it was just tracking the broader market.
Test 2: Monte Carlo resampling
A different commenter pointed out that a single walk-forward result still isn’t immune to the layout of the data. So I cut the three years into monthly blocks, shuffled the block order 200 times, and re-ran the fixed strategy on a fresh out-of-sample split each time.
My original result landed exactly on the 50th percentile of the resulting distribution — not a lucky draw, not an unlucky one. But the distribution itself was wide: 5th percentile was -40%, 95th percentile was +102%. The center is solid; the tails are real.
Test 3: Alpha and beta against buy-and-hold
Someone then asked the question I’d been dodging: no beta check, no alpha, no benchmark — so how do I know this isn’t just tracking the underlying asset?
I ran the regression against simply holding ETH over the same period. Beta came out at 0.117 — largely uncorrelated with ETH’s own price swings. Alpha was +18.4% annualized. The number that mattered most wasn’t the regression, though — it was the direct comparison: buy-and-hold ETH returned -6.6% over this window. The strategy returned +79.2%. It made money while the benchmark lost money, in the same period, which is a different and stronger claim than “beat the market during a bull run.”
Test 4: A t-test on that alpha
Then the same person came back with the sharper question: a positive alpha estimate doesn’t mean much without knowing if it’s distinguishable from zero, especially after a 324-combination search that will always produce some inflated numbers by chance.
I ran the t-test on the daily alpha residuals — 1,197 observations, way more statistical power than the 19 trades I’d been working with in earlier tests. The result: t-stat of 1.45. Below the 1.96 threshold for 95% confidence. Not significant.
Where this leaves things
Four tests, four different questions, and they don’t collapse into one clean verdict:
- Consistent across time: yes (walk-forward)
- Not a fluke of resampling: yes (Monte Carlo)
- Not just market exposure: probably (low beta, positive alpha point estimate, outperformance in a down market)
- Statistically distinguishable from zero: no, not yet (t-stat 1.45 against a 1.96 bar)
That last line is the honest ceiling on what I can claim right now. The strategy shows every sign of a real, modest edge — but with crypto’s daily noise, even ~1,200 observations isn’t enough to push it over a standard significance threshold. That’s not a failure of the strategy; it might just be the honest cost of testing anything in a market this volatile.
What I’m doing with this
Nothing to the live settings. Re-optimizing again on the same three years of history would just reintroduce the exact selection bias problem from the first post, one layer deeper. The only clean test left is time itself — new bars that didn’t exist when any parameter was chosen. That’s what the bot is generating every day it keeps running, and it’s the only evidence from here that actually counts as fresh.
If you’re validating your own strategy and only run one of these four tests, my honest suggestion is: run the last one. Consistency and resampling checks feel reassuring, but they can both come back clean on a strategy that still isn’t statistically distinguishable from noise. The t-test is the one that keeps you honest.
