# OOS Degradation Monitor — Power & Detection-Latency Study (2026-05-30)

**Type:** research note (NON-alpha). Characterizes the forward OOS degradation
monitor (`backtest/oos_monitor.py`) so its verdicts are trustworthy. This study
*found and fixed a real design flaw in the monitor's own binding rule.*

Reproduce: `oos_monitor.power_curve(load_in_sample_returns(), sharpes, Ns)`.

## The flaw: the naive rule cried wolf ~40% of the time

The monitor's first binding rule was: **DEGRADED when the live Sharpe point
estimate < the in-sample Sharpe bootstrap CI lower bound (0.479).** That ignores
the *live* estimate's own (large) sampling error. A live Sharpe measured over N
days has SE ≈ √(252/N) annualized — at N=60 that is ~2.0. So even when live
performance is *truly identical* to in-sample (Sharpe 0.93), the live point
estimate dips below 0.479 by pure noise very often.

Simulated false-alarm rate of the naive rule (live truly == in-sample):

| N (days) | 30 | 60 | 90 | 126 | 252 | 504 |
|----------|----|----|----|-----|-----|-----|
| P(false DEGRADED) | 43% | 42% | 40% | 38% | 33% | 27% |

A 27–43% false-alarm rate is a broken judge: it would scream "edge is dead"
roughly every other quarter on a perfectly healthy strategy. This is exactly the
"cry wolf" failure the brief warned against — and the naive rule's power was
barely above this floor, so it had almost no discriminating value.

## The fix: account for the live sample's own uncertainty

**Corrected binding rule: DEGRADED only when the live Sharpe's bootstrap UPPER
95% bound is below the in-sample Sharpe point estimate** — i.e. live
underperformance exceeds its own sampling error (a ~2.5% one-sided false-alarm
target). Implemented in `evaluate_degradation`; the report now surfaces the live
95% upper bound as the trigger.

## Operating characteristics of the corrected rule

P(monitor flags DEGRADED) by true live Sharpe × live days (in-sample Sharpe 0.926):

| true Sharpe \ N | 60 | 126 | 252 | 504 | 756 | 1260 |
|-----------------|----|-----|-----|-----|-----|------|
| **0.93 (==IS)** | **4%** | **3%** | **3%** | **3%** | **4%** | **4%** |
| 0.6 | 7% | 4% | 6% | 8% | 7% | 10% |
| 0.4 | 8% | 5% | 9% | 11% | 14% | 18% |
| 0.2 | 9% | 7% | 14% | 19% | 26% | 35% |
| 0.0 | 10% | 10% | 16% | 26% | 40% | 55% |
| -0.5 | 16% | 16% | 30% | 55% | 74% | 91% |

(252 days ≈ 1 year, 1260 ≈ 5 years.)

## Caveat on the numbers

These power/false-alarm figures simulate live streams as **IID Normal** draws
calibrated to the in-sample daily vol. Real live returns are fatter-tailed and
mildly autocorrelated, which widens the live bootstrap CI — so the real
false-alarm rate is, if anything, *lower* (more conservative), and the real power
*lower* still. The qualitative conclusion (controlled false alarms, low power,
catastrophe-only detection) is robust to this; the exact cell values are not
gospel. The production rule itself uses the live sample's own bootstrap (no Normal
assumption), so it adapts to the true return shape.

## The honest, sobering conclusion

- **False alarms are now controlled (~3–4%)** vs ~40% before — a real fix.
- **The price is low power, and that is unavoidable.** Sharpe is a very noisy
  estimator. To detect a true *halving* of the edge (0.93 → ~0.4) the monitor
  needs years and still only reaches ~18% power at 5 years. Only a **severe,
  sustained collapse** (Sharpe → ≤0 over multiple years) is detected reliably
  (negative Sharpe: 74% at 3y, 91% at 5y).
- Therefore the monitor is a **catastrophe backstop, not an early-warning
  system.** A passing `OK` does NOT certify the edge — it means "no collapse
  detectable above noise yet." This is now stated in the monitor's power note.
- **Implication for the live A/B:** subtle edge decay is statistically
  undetectable in any reasonable horizon from returns alone. The live A/B should
  be judged on *effect sizes and economic priors*, with this monitor only as a
  hard tripwire against catastrophic, sustained failure. Detecting subtle decay
  needs either far longer samples or higher-frequency / cross-sectional signals
  than monthly equity marks provide.

## Why keep MIN_OOS_DAYS = 60

With the corrected rule the false-alarm rate is controlled at every N, so the
60-day gate is no longer a false-alarm guard — it is an anti-over-interpretation
guard (below ~3 months the verdict carries almost no information either way).
INSUFFICIENT_DATA remains the honest verdict for the current live sample (~37
trimmed days).