# Null Calibration of the Falsification Engine — Pre-Registration (2026-08-01)

**Status: BINDING PRE-REGISTRATION, append-only after today.** Written and
committed BEFORE any calibration run exists (panel item C2, quant panel report
2026-08-01, finding 8). The runner (`scripts/run_null_calibration.sh`) refuses
to execute unless this file is present, and every result JSON records this
file's SHA-256 — so a result that predates or bypasses the registration is
detectable, not deniable.

**This is calibration only.** No trading decision, no strategy verdict, and no
threshold changes hang on the outcome. **ALL existing verdicts stand regardless
of what these runs print** — the momentum baseline numbers, every recorded
kill (including tranching 2026-06-10, subject only to §6's narrow reopening
clause below, which grants *registration eligibility*, never a verdict
change), and every SHIP-gate rejection. What changes afterward is only how
future numbers are *quoted*: once the null distribution exists, future PBO /
IS-OOS-correlation thresholds are quoted **as percentiles of the null** next
to their absolute values.

---

## 1. The problem, from first principles

The shop's central kill statistic is **PBO** — the *probability of backtest
overfitting*: across the 15 CPCV paths (train/test splits), how often do the
paths that looked best in-sample land in the bottom half out-of-sample. The
implementation (`backtest/cpcv.py:_compute_pbo`) is a **nonstandard
single-config rank variant**: with 15 paths, the top-IS half holds 8 paths, so
PBO moves in **quanta of 1/8 = 12.5 percentage points** — the boundary between
"eligible" and "killed" can be ONE path flipping sides. The companion
statistic, **IS-OOS correlation** (do splits that look good in-sample also
look good out-of-sample), is partly *arithmetic* for episodic strategies: a
hot episode assigned to a test block is, by construction, missing from that
path's train metric, pushing the correlation negative with zero overfitting
involved.

Nobody has ever measured what this harness prints for a book with **no
cross-sectional skill**. Every threshold we quote (PBO 50% "eligible", 75%
"overfit", corr −0.70 "warning") is an absolute number on an uncalibrated
scale. This memo pins the calibration before the first run, so the reading
cannot be shopped after the numbers exist.

## 2. The pinned harness

Identical to the production standard-mode CPCV (`thales cpcv`), varying
nothing but the strategy plugged in:

- `run_cpcv` with **n_groups=6, k_test=2 (15 paths), purge_days=252,
  embargo_days=5** — the keystone-pinned `cpcv:` block of
  `config/settings.yaml`.
- Data: the **full frozen `data/raw` panel**, standard membership mode (no
  survivorship filter) — exactly what `thales cpcv` uses. Frozen inputs make
  every run deterministic and reproducible.
- Base config: `config/settings.yaml` as committed, with ONLY the per-config
  overrides listed in §3. In particular the production construction overlays
  (half-Kelly, vol target + VIX scaling, position/sector caps, kill-switch,
  turnover cap, flat 10 bps costs, monthly stock selection) all stay ON — the
  point is to measure what the WHOLE apparatus prints for no-skill books, not
  what a stripped engine prints.
- Worker count is free (paths are independent; results are
  worker-count-invariant).

Honest note on the survivor-biased panel: `data/raw` is survivor-heavy, so the
nulls' *absolute* Sharpes will be inflated. That is irrelevant here — PBO and
IS-OOS correlation are rank/relative statistics *within* a config across
paths; the calibration question is about the statistic's arithmetic, not the
nulls' returns. (A survivorship-free replication on the golden store is a
legitimate follow-up, not part of this registration.)

## 3. The pinned null set (24 configs)

All selection is done by `RandomBookStrategy` (`strategy/random_book.py`,
registered as `random_book`) — a **calibration instrument**: selection is a
deterministic seeded hash of (seed, symbol, calendar month), so it cannot read
prices, and a fresh uniform-random book is drawn each month ("monthly
reselection", matching production's selection cadence). Equal weight at the
strategy layer (random names give no basis for anything else); the
construction overlays then apply exactly as in production
(`strategy.construction.weighting` stays `"hrp"` so Kelly integrates as the
same portfolio-level scalar it is for the momentum book — the identical
overlay code path).

| id | class | overrides on top of settings.yaml |
|---|---|---|
| `spy_buyhold` | (a) single-asset market reference | `strategy.name=random_book`, `strategy.random_book.seed=0`, `strategy.random_book.book_size=1`, `strategy.random_book.symbols=["SPY"]`, `risk.max_single_position_pct=1.0`, `risk.max_sector_pct=1.0` |
| `random_seed01` … `random_seed20` | (b) 20 seeded random 50-name books | `strategy.name=random_book`, `strategy.random_book.seed=<1..20>` (book_size 50 = the production sleeve size, min_history_days 63) |
| `composite_seed01` … `composite_seed03` | (c) 3 timing-averaged composites | the (b) config for seeds 1, 2, 3 **plus** `rebalance.tranche_days=[1,8,15,22]` |

Pinned details:

- **(a)** is SPY buy-and-hold *under the harness overlays*: a 1-name book
  restricted to SPY, "reselected" monthly (always SPY). The two risk-cap
  overrides are the ONE documented deviation from the production config: a
  10% single-position cap on a single-asset book would silently turn "SPY"
  into "10% SPY + cash" — a cross-sectional constraint with no meaning at
  book size 1. Everything else (Kelly, vol target, kill-switch, costs)
  applies. Its role: the cleanest probe of the **episodic-correlation
  arithmetic** — a deterministic market book has zero selection skill and
  zero selection variance, so whatever IS-OOS correlation it prints is pure
  window arithmetic.
- **(b)** seeds are **1..20 exactly**, pinned here; 50 names uniformly at
  random each month from the eligible universe (a name is eligible on a date
  if it has a price that day and ≥ 63 prior trading days of history —
  `min_history_days=63`, matching the engine's own history floor). These 20
  runs ARE "the null PBO distribution" wherever this memo says that phrase.
- **(c)** uses the SAME seeds 1–3, so each composite trades the *same monthly
  book sequence* as its (b) twin, averaged across the engine's pre-registered
  tranching mode: 4 overlapping sleeves on monthly schedules staggered at
  days [1, 8, 15, 22] of each month, 1/4 capital each, no cross-sleeve
  transfers — the δ=0 arm of the 2026-06-10 tranching experiment, exactly.
  Holding selection fixed and varying ONLY timing-averaging makes (c)-vs-(b)
  the clean probe of whether the statistic punishes timing-averaged books
  per se — which is what decides whether the tranching kill was partly a
  statistic artifact.

These 24 runs are calibration instruments, **not strategy trials**: nothing
will ever be selected from among them to trade, so they carry no
multiple-testing debt and must never enter the DSR trial count (they write to
`results/diagnostics/null_calibration/`, which the trial counter does not
read; recorded here so C3's registry work inherits the exclusion rationale).

## 4. The pinned refutation statistic (exact)

Let `PBO_b = {PBO of the 20 (b) runs}` and `corr_b = {IS-OOS Sharpe
correlation of the 20 (b) runs}` (the `is_oos_correlation` field of each
result JSON). "Centers" is pinned as the **median**.

> **The artifact hypothesis is REFUTED iff**
> `|median(PBO_b) − 50%| ≤ one path-quantum` **AND** `median(|corr_b|) < 0.2`,
> where one path-quantum = `1 / ceil(n_paths/2)` = **12.5 pp at the expected
> 15 paths** (if any run realizes fewer paths, the quantum is restated
> mechanically from that run's realized count; the summariser prints the
> per-run quanta and uses the largest).

- **REFUTED** → the harness prints ~50%/~0 for no-skill books: the absolute
  thresholds are **vindicated** as-is, and **the tranching door closes
  permanently** (the 2026-06-10 kill stands with no artifact caveat — never
  relitigate).
- **NOT REFUTED** (the null centers away from 50% or carries structural
  |corr| ≥ 0.2) → no verdict changes; from then on every quoted PBO / corr
  carries its null percentile next to the absolute number, and future
  batteries state their thresholds in null percentiles at registration time.

**The tranching reopening clause (§6 of item C2), pinned:** only a
**systematic (c)-vs-(b) PBO gap** reopens tranching — defined as: all 3
same-seed paired differences `PBO(composite_s) − PBO(random_s)` **> 0**
(sign-unanimous) AND their mean **≥ one path-quantum (12.5 pp)**. If that
fires, tranching v2 earns **eligibility to write a NEW hash-frozen
registration — never to ship**, and never a retroactive change to the
2026-06-10 verdict. Anything less than sign-unanimity + a quantum-sized mean
is noise at n=3 and closes the door with the main refutation statistic.

## 5. Resolution limits (carried in print, per the red team)

- **20 draws** resolve the null distribution to ~5 pp percentile granularity;
  a future "the live book is at the Xth null percentile" claim can never be
  sharper than that.
- Each draw's PBO lives on a **12.5 pp lattice** (15 paths); the median of 20
  lattice values inherits that coarseness. The refutation bar is therefore
  exactly one lattice step — the finest honest bar this harness allows.
- The (c)-vs-(b) probe has **n=3 pairs**: it is direction-plus-magnitude
  screening, not significance testing — which is why its only possible payoff
  is *registration eligibility* for a properly-powered v2 probe.
- SPY (a) is a single config: it is REPORTED as the episodic-arithmetic
  demonstration and sits **outside** the refutation statistic (pinned here so
  no later reading can quietly swap which subset is "the distribution").

## 6. Procedure (mechanical)

1. This memo is committed BEFORE any run; the runner asserts the file exists
   and stamps its SHA-256 into every result JSON.
2. `scripts/run_null_calibration.sh` runs the 24 configs **sequentially under
   `nice`** (each ≈ a full 15-path CPCV; the batch is hours — a weekend of
   home compute). Each run writes
   `results/diagnostics/null_calibration/<id>.json`; existing files are
   skipped, so the batch is resumable.
3. `scripts/summarize_null_calibration.py` then evaluates §4 **mechanically**
   — the REFUTED/NOT-REFUTED line and the tranching-door line are computed,
   not narrated — and writes `summary.md` beside the JSONs (distribution
   table, per-run quanta, paired (c)-vs-(b) table, this memo's hash, and the
   §5 resolution limits verbatim).
4. Interpretation beyond §4/§5 (if any) is appended to THIS memo with a date,
   below this line — the pinned text above is never edited.

*Pinned 2026-08-01. Panel-Item: C2.*

---

## RESULTS APPEND — 2026-08-01 (batch complete; summary: results/diagnostics/null_calibration/summary.md)

24 pinned runs (memo SHA verified consistent across all). The §4 refutation
statistic, evaluated mechanically:

- (b) random ensemble n=20: **PBO median 0.500 — |median − 50%| = 0.000,
  centered within one path-quantum → the PBO CENTER is vindicated.**
- median |IS-OOS corr| on NO-SKILL books = **0.443** vs the 0.2 bar →
  **artifact hypothesis NOT REFUTED**: strongly negative IS-OOS correlation
  is what this harness prints for books with no skill at all (range
  −0.02..−0.80 across seeds). A negative corr is NOT evidence a strategy is
  overfit; it is substantially the episodic-arithmetic artifact §2 described.
- **Convention change (pre-registered here):** all existing verdicts stand;
  from now on quoted PBO / corr values carry their null percentiles
  (~5 pp granularity — 20 draws; carried in print).

**Tranching door (§4, probe c): CLOSED.** Same-seed composite-vs-random PBO
diffs +0.375 / +0.125 / −0.125 — not sign-unanimous, mean +0.125 = exactly
one quantum. No systematic punishment of timing-averaged books. The
2026-06-10 tranching kill stands permanently; no v2 registration is earned.

Independent corroboration: the C4 purge grid (same night) measured the
Kelly-ledger purge leak pushing corr −0.281 → −0.699 at the production
window — two instruments, one conclusion: **the corr headline was mostly
the instrument, not the strategy.**