Null Calibration of the Falsification Engine — Pre-Registration (2026-08-01)

Status: BINDING PRE-REGISTRATION, append-only after today. Written and committed BEFORE any calibration run exists (panel item C2, quant panel report 2026-08-01, finding 8). The runner (scripts/run_null_calibration.sh) refuses to execute unless this file is present, and every result JSON records this file's SHA-256 — so a result that predates or bypasses the registration is detectable, not deniable.

This is calibration only. No trading decision, no strategy verdict, and no threshold changes hang on the outcome. ALL existing verdicts stand regardless of what these runs print — the momentum baseline numbers, every recorded kill (including tranching 2026-06-10, subject only to §6's narrow reopening clause below, which grants registration eligibility, never a verdict change), and every SHIP-gate rejection. What changes afterward is only how future numbers are quoted: once the null distribution exists, future PBO / IS-OOS-correlation thresholds are quoted as percentiles of the null next to their absolute values.

1. The problem, from first principles

The shop's central kill statistic is PBO — the probability of backtest overfitting: across the 15 CPCV paths (train/test splits), how often do the paths that looked best in-sample land in the bottom half out-of-sample. The implementation (backtest/cpcv.py:_compute_pbo) is a nonstandard single-config rank variant: with 15 paths, the top-IS half holds 8 paths, so PBO moves in quanta of 1/8 = 12.5 percentage points — the boundary between "eligible" and "killed" can be ONE path flipping sides. The companion statistic, IS-OOS correlation (do splits that look good in-sample also look good out-of-sample), is partly arithmetic for episodic strategies: a hot episode assigned to a test block is, by construction, missing from that path's train metric, pushing the correlation negative with zero overfitting involved.

Nobody has ever measured what this harness prints for a book with no cross-sectional skill. Every threshold we quote (PBO 50% "eligible", 75% "overfit", corr −0.70 "warning") is an absolute number on an uncalibrated scale. This memo pins the calibration before the first run, so the reading cannot be shopped after the numbers exist.

2. The pinned harness

Identical to the production standard-mode CPCV (thales cpcv), varying nothing but the strategy plugged in:

• run_cpcv with n_groups=6, k_test=2 (15 paths), purge_days=252, embargo_days=5 — the keystone-pinned cpcv: block of config/settings.yaml.
• Data: the full frozen data/raw panel, standard membership mode (no survivorship filter) — exactly what thales cpcv uses. Frozen inputs make every run deterministic and reproducible.
• Base config: config/settings.yaml as committed, with ONLY the per-config overrides listed in §3. In particular the production construction overlays (half-Kelly, vol target + VIX scaling, position/sector caps, kill-switch, turnover cap, flat 10 bps costs, monthly stock selection) all stay ON — the point is to measure what the WHOLE apparatus prints for no-skill books, not what a stripped engine prints.
• Worker count is free (paths are independent; results are worker-count-invariant).

Honest note on the survivor-biased panel: data/raw is survivor-heavy, so the nulls' absolute Sharpes will be inflated. That is irrelevant here — PBO and IS-OOS correlation are rank/relative statistics within a config across paths; the calibration question is about the statistic's arithmetic, not the nulls' returns. (A survivorship-free replication on the golden store is a legitimate follow-up, not part of this registration.)

3. The pinned null set (24 configs)

All selection is done by RandomBookStrategy (strategy/random_book.py, registered as random_book) — a calibration instrument: selection is a deterministic seeded hash of (seed, symbol, calendar month), so it cannot read prices, and a fresh uniform-random book is drawn each month ("monthly reselection", matching production's selection cadence). Equal weight at the strategy layer (random names give no basis for anything else); the construction overlays then apply exactly as in production (strategy.construction.weighting stays "hrp" so Kelly integrates as the same portfolio-level scalar it is for the momentum book — the identical overlay code path).

Pinned details:

• (a) is SPY buy-and-hold under the harness overlays: a 1-name book restricted to SPY, "reselected" monthly (always SPY). The two risk-cap overrides are the ONE documented deviation from the production config: a 10% single-position cap on a single-asset book would silently turn "SPY" into "10% SPY + cash" — a cross-sectional constraint with no meaning at book size 1. Everything else (Kelly, vol target, kill-switch, costs) applies. Its role: the cleanest probe of the episodic-correlation arithmetic — a deterministic market book has zero selection skill and zero selection variance, so whatever IS-OOS correlation it prints is pure window arithmetic.
• (b) seeds are 1..20 exactly, pinned here; 50 names uniformly at random each month from the eligible universe (a name is eligible on a date if it has a price that day and ≥ 63 prior trading days of history — min_history_days=63, matching the engine's own history floor). These 20 runs ARE "the null PBO distribution" wherever this memo says that phrase.
• (c) uses the SAME seeds 1–3, so each composite trades the same monthly book sequence as its (b) twin, averaged across the engine's pre-registered tranching mode: 4 overlapping sleeves on monthly schedules staggered at days [1, 8, 15, 22] of each month, 1/4 capital each, no cross-sleeve transfers — the δ=0 arm of the 2026-06-10 tranching experiment, exactly. Holding selection fixed and varying ONLY timing-averaging makes (c)-vs-(b) the clean probe of whether the statistic punishes timing-averaged books per se — which is what decides whether the tranching kill was partly a statistic artifact.

These 24 runs are calibration instruments, not strategy trials: nothing will ever be selected from among them to trade, so they carry no multiple-testing debt and must never enter the DSR trial count (they write to results/diagnostics/null_calibration/, which the trial counter does not read; recorded here so C3's registry work inherits the exclusion rationale).

4. The pinned refutation statistic (exact)

Let PBO_b = {PBO of the 20 (b) runs} and corr_b = {IS-OOS Sharpe correlation of the 20 (b) runs} (the is_oos_correlation field of each result JSON). "Centers" is pinned as the median.

The artifact hypothesis is REFUTED iff |median(PBO_b) − 50%| ≤ one path-quantum AND median(|corr_b|) < 0.2, where one path-quantum = 1 / ceil(n_paths/2) = 12.5 pp at the expected 15 paths (if any run realizes fewer paths, the quantum is restated mechanically from that run's realized count; the summariser prints the per-run quanta and uses the largest).

• REFUTED → the harness prints ~50%/~0 for no-skill books: the absolute thresholds are vindicated as-is, and the tranching door closes permanently (the 2026-06-10 kill stands with no artifact caveat — never relitigate).
• NOT REFUTED (the null centers away from 50% or carries structural |corr| ≥ 0.2) → no verdict changes; from then on every quoted PBO / corr carries its null percentile next to the absolute number, and future batteries state their thresholds in null percentiles at registration time.

The tranching reopening clause (§6 of item C2), pinned: only a systematic (c)-vs-(b) PBO gap reopens tranching — defined as: all 3 same-seed paired differences PBO(composite_s) − PBO(random_s) > 0 (sign-unanimous) AND their mean ≥ one path-quantum (12.5 pp). If that fires, tranching v2 earns eligibility to write a NEW hash-frozen registration — never to ship, and never a retroactive change to the 2026-06-10 verdict. Anything less than sign-unanimity + a quantum-sized mean is noise at n=3 and closes the door with the main refutation statistic.

5. Resolution limits (carried in print, per the red team)

• 20 draws resolve the null distribution to ~5 pp percentile granularity; a future "the live book is at the Xth null percentile" claim can never be sharper than that.
• Each draw's PBO lives on a 12.5 pp lattice (15 paths); the median of 20 lattice values inherits that coarseness. The refutation bar is therefore exactly one lattice step — the finest honest bar this harness allows.
• The (c)-vs-(b) probe has n=3 pairs: it is direction-plus-magnitude screening, not significance testing — which is why its only possible payoff is registration eligibility for a properly-powered v2 probe.
• SPY (a) is a single config: it is REPORTED as the episodic-arithmetic demonstration and sits outside the refutation statistic (pinned here so no later reading can quietly swap which subset is "the distribution").

6. Procedure (mechanical)

1. This memo is committed BEFORE any run; the runner asserts the file exists and stamps its SHA-256 into every result JSON.
2. scripts/run_null_calibration.sh runs the 24 configs sequentially under nice (each ≈ a full 15-path CPCV; the batch is hours — a weekend of home compute). Each run writes results/diagnostics/null_calibration/<id>.json; existing files are skipped, so the batch is resumable.
3. scripts/summarize_null_calibration.py then evaluates §4 mechanically — the REFUTED/NOT-REFUTED line and the tranching-door line are computed, not narrated — and writes summary.md beside the JSONs (distribution table, per-run quanta, paired (c)-vs-(b) table, this memo's hash, and the §5 resolution limits verbatim).
4. Interpretation beyond §4/§5 (if any) is appended to THIS memo with a date, below this line — the pinned text above is never edited.

Pinned 2026-08-01. Panel-Item: C2.

RESULTS APPEND — 2026-08-01 (batch complete; summary: results/diagnostics/null_calibration/summary.md)

24 pinned runs (memo SHA verified consistent across all). The §4 refutation statistic, evaluated mechanically:

• (b) random ensemble n=20: PBO median 0.500 — |median − 50%| = 0.000, centered within one path-quantum → the PBO CENTER is vindicated.
• median |IS-OOS corr| on NO-SKILL books = 0.443 vs the 0.2 bar → artifact hypothesis NOT REFUTED: strongly negative IS-OOS correlation is what this harness prints for books with no skill at all (range −0.02..−0.80 across seeds). A negative corr is NOT evidence a strategy is overfit; it is substantially the episodic-arithmetic artifact §2 described.
• Convention change (pre-registered here): all existing verdicts stand; from now on quoted PBO / corr values carry their null percentiles (~5 pp granularity — 20 draws; carried in print).

Tranching door (§4, probe c): CLOSED. Same-seed composite-vs-random PBO diffs +0.375 / +0.125 / −0.125 — not sign-unanimous, mean +0.125 = exactly one quantum. No systematic punishment of timing-averaged books. The 2026-06-10 tranching kill stands permanently; no v2 registration is earned.

Independent corroboration: the C4 purge grid (same night) measured the Kelly-ledger purge leak pushing corr −0.281 → −0.699 at the production window — two instruments, one conclusion: the corr headline was mostly the instrument, not the strategy.