eagL / chronological holdouts

Why chronological holdouts matter in prediction-market backtests

Every backtest is a claim that the past predicts the future. A randomly split backtest quietly makes a stronger claim: that the future predicts the past. This note walks through the leak, why prediction markets make it worse, and the regime that closes it.

The mechanics of the leak

Shuffle your samples into train and test sets and each test point sits surrounded, in time, by training points. The model has seen Tuesday 2pm and Tuesday 4pm; you test it on Tuesday 3pm. Markets are autocorrelated at exactly these scales, so the model gets credit for interpolating a regime it was shown, which is a skill that does not exist at deployment time, where every trade happens after all the data you have.

Why prediction markets leak worse

Event contracts amplify the problem in two ways. Contracts cluster: fourteen crypto strike series on the same asset resolve off the same underlying path, so a random split scatters near-duplicates of one event across both sets. And regimes are discrete: a rule tuned during one volatility regime tests brilliantly on shuffled samples of that regime and then meets a different one on Monday. In both cases the random split converts one lucky period into what looks like many independent confirmations.

The chronological regime

Split time in order: tune on the early window, hold out the late one. The ratio matters less than the discipline around it; 60/40 leaves a test window long enough to mean something. Touch the held-out window once. Test, look, stop. The second evaluation on the same holdout begins fitting to it, just with extra steps, and after a dozen iterations the holdout has become training data with a better reputation.

Read the drop, not the number

The honest output of a holdout is two numbers and the distance between them. A strategy earning 8% in sample and 7% out of sample generalizes. One earning 8% and 1% is a 1% strategy that memorized its decade. The drop also budgets your sweep: the more parameter combinations you tried, the larger the drop you should expect, which is why a cap on the sweep is not conservatism but arithmetic.

How eagL applies this

eagL splits chronologically at 60/40, evaluates the holdout exactly once, shows both numbers with the drop, and caps parameter sweeps at a few hundred combinations. The surrounding gates, minimum detections, fills, outcomes and days, are specified in the methodology, and the product-level argument is on the backtesting page.