OOS Degradation Monitor — Power & Detection-Latency Study (2026-05-30)
Type: research note (NON-alpha). Characterizes the forward OOS degradation
monitor (backtest/oos_monitor.py) so its verdicts are trustworthy. This study
found and fixed a real design flaw in the monitor's own binding rule.
Reproduce: oos_monitor.power_curve(load_in_sample_returns(), sharpes, Ns).
The flaw: the naive rule cried wolf ~40% of the time
The monitor's first binding rule was: DEGRADED when the live Sharpe point estimate < the in-sample Sharpe bootstrap CI lower bound (0.479). That ignores the live estimate's own (large) sampling error. A live Sharpe measured over N days has SE ≈ √(252/N) annualized — at N=60 that is ~2.0. So even when live performance is truly identical to in-sample (Sharpe 0.93), the live point estimate dips below 0.479 by pure noise very often.
Simulated false-alarm rate of the naive rule (live truly == in-sample):
| N (days) | 30 | 60 | 90 | 126 | 252 | 504 |
|---|---|---|---|---|---|---|
| P(false DEGRADED) | 43% | 42% | 40% | 38% | 33% | 27% |
A 27–43% false-alarm rate is a broken judge: it would scream "edge is dead" roughly every other quarter on a perfectly healthy strategy. This is exactly the "cry wolf" failure the brief warned against — and the naive rule's power was barely above this floor, so it had almost no discriminating value.
The fix: account for the live sample's own uncertainty
Corrected binding rule: DEGRADED only when the live Sharpe's bootstrap UPPER
95% bound is below the in-sample Sharpe point estimate — i.e. live
underperformance exceeds its own sampling error (a ~2.5% one-sided false-alarm
target). Implemented in evaluate_degradation; the report now surfaces the live
95% upper bound as the trigger.
Operating characteristics of the corrected rule
P(monitor flags DEGRADED) by true live Sharpe × live days (in-sample Sharpe 0.926):
| true Sharpe \ N | 60 | 126 | 252 | 504 | 756 | 1260 |
|---|---|---|---|---|---|---|
| 0.93 (==IS) | 4% | 3% | 3% | 3% | 4% | 4% |
| 0.6 | 7% | 4% | 6% | 8% | 7% | 10% |
| 0.4 | 8% | 5% | 9% | 11% | 14% | 18% |
| 0.2 | 9% | 7% | 14% | 19% | 26% | 35% |
| 0.0 | 10% | 10% | 16% | 26% | 40% | 55% |
| -0.5 | 16% | 16% | 30% | 55% | 74% | 91% |
(252 days ≈ 1 year, 1260 ≈ 5 years.)
Caveat on the numbers
These power/false-alarm figures simulate live streams as IID Normal draws calibrated to the in-sample daily vol. Real live returns are fatter-tailed and mildly autocorrelated, which widens the live bootstrap CI — so the real false-alarm rate is, if anything, lower (more conservative), and the real power lower still. The qualitative conclusion (controlled false alarms, low power, catastrophe-only detection) is robust to this; the exact cell values are not gospel. The production rule itself uses the live sample's own bootstrap (no Normal assumption), so it adapts to the true return shape.
The honest, sobering conclusion
- False alarms are now controlled (~3–4%) vs ~40% before — a real fix.
- The price is low power, and that is unavoidable. Sharpe is a very noisy estimator. To detect a true halving of the edge (0.93 → ~0.4) the monitor needs years and still only reaches ~18% power at 5 years. Only a severe, sustained collapse (Sharpe → ≤0 over multiple years) is detected reliably (negative Sharpe: 74% at 3y, 91% at 5y).
- Therefore the monitor is a catastrophe backstop, not an early-warning
system. A passing
OKdoes NOT certify the edge — it means "no collapse detectable above noise yet." This is now stated in the monitor's power note. - Implication for the live A/B: subtle edge decay is statistically undetectable in any reasonable horizon from returns alone. The live A/B should be judged on effect sizes and economic priors, with this monitor only as a hard tripwire against catastrophic, sustained failure. Detecting subtle decay needs either far longer samples or higher-frequency / cross-sectional signals than monthly equity marks provide.
Why keep MIN_OOS_DAYS = 60
With the corrected rule the false-alarm rate is controlled at every N, so the 60-day gate is no longer a false-alarm guard — it is an anti-over-interpretation guard (below ~3 months the verdict carries almost no information either way). INSUFFICIENT_DATA remains the honest verdict for the current live sample (~37 trimmed days).