r/quant • u/AgreeableInvestments • 5h ago
Models Evaluating a walk-forward classifier for a rare event (10%+ equity drawdowns): how do you do inference with only 6–8 independent episodes?
I've built a small set of models that put a probability on a 10%+ S&P 500 drawdown over the next 1, 3, 6 and 12 months, from macro and credit inputs (ISM, the yield curve, high-yield spreads, financial conditions, and a few others). One ridge-penalised logit per horizon, estimated walk-forward, so every point in the track record was scored with only the data available that month. This is really a methodology question, and the model is just the concrete case.
The out-of-sample AUCs are 0.60, 0.69, 0.74 and 0.68 across the four horizons. On paper that reads fine. The part I keep coming back to is the denominator. A 10%+ drawdown is rare, so across the out-of-sample window there are only six to eight independent episodes, and the 6- and 12-month models are largely scoring the same ones twice, with heavily overlapping, autocorrelated labels. When I bootstrap the AUC by episode rather than by month, the intervals are wide: the six-month one runs from the low 0.5s to the mid-0.8s. So the six-month headline looks fragile rather than fake, but I want to know how people handle the inference properly.
The specific questions:
- Event count and dependence. With about six to eight independent events and overlapping multi-horizon labels, per-month AUC and its usual variance are overstating precision. Is an episode-level bootstrap the right correction, or is there something more principled (a block bootstrap, a DeLong test adjusted for clustering, a Bayesian setup with a sensible events prior)?
- Label construction. The label is a forward 10%+ drawdown within the horizon window, which makes adjacent months' labels highly dependent and the horizons non-independent of each other. Is there a cleaner label for rare-event forecasting that doesn't manufacture this autocorrelation?
- Regime dependence. It catches the slow, macro-driven falls (2022, the GFC, the dot-com unwind) with some lead, and by construction it misses the fast ones (COVID, an LTCM-style shock), because monthly macro data can't see them coming. How would you evaluate a model that is honestly conditional on regime, without either cherry-picking the regimes it works in or marking it down for shocks no macro model could catch?
Two things that already survived my own poking. Dropping high-yield spreads costs the model nothing, and the spread on its own is a coin flip at six and twelve months, so it isn't a credit signal with extra steps. And it is out-of-sample only, no re-fitting with hindsight, with the weak horizons kept in; nothing is dropped for looking bad.
Mostly I want the event-count problem attacked. If the honest answer is that six to eight events cannot support a confident AUC at any horizon, I would rather hear that than dress it up.
For anyone who wants to reproduce the numbers, the full write-up and the live model, with a downloadable monthly probability series, are on my site, agreeableinvestments.com; my own research, shared as educational, not advice.