2026-08-07 — the skew capture lost a day and gained a contaminated one
Class: INCIDENT (data integrity, forward capture). Found by: the daily audit, from the ground-truth side. Nothing alarmed.
What happened
The 2026-08-06 runner-loss outage (audit PR #90) displaced the Skew Snapshot
job. Its cron is 0 19 * * 1-5; the run was created at 2026-08-07T00:42Z,
about 5.7 hours late — past UTC midnight.
thales snapshot-skew stamps its output with the runner's UTC calendar date
(cli.py: snapshot_skew(..., today=_dt.date.today())). So the run that was
supposed to capture Thursday filed everything under Friday:
| stream | 2026-08-06 | 2026-08-07 |
|---|---|---|
| options_skew | absent | 891 rows from the displaced run + 5 genuine |
| options_chains | absent | overwritten by Friday's run (genuine) |
| options_iv_term | absent | overwritten by Friday's run (genuine) |
| shortability | absent | displaced run's file, never refreshed |
Two distinct failure shapes, from the same cause:
-
A permanent hole at 2026-08-06. Free point-in-time IV does not exist retroactively — this is the failure mode the whole stream is classified TIME-GATED to prevent.
-
Contamination of 2026-08-07, which is worse than a hole because it is invisible.
snapshot_skewbuilds adoneset of symbols already recorded fortodayand skips them (a resume guard for partial failures). Friday's genuine run therefore found 891 symbols "already done" and appended only the 5 that were missing. Verified: all 891 rows are byte-identical to the displaced run's write, and theirdtevalues are computed against 2026-08-07, so they carry Thursday's quotes under Friday's label with Friday's arithmetic.
options_chains and options_iv_term write whole-file per day, so Friday's
run overwrote them — those two are genuinely Friday's. shortability was not
rewritten at all. The four streams from one run now disagree about what day
they hold.
Why nothing fired
thales capturesjudges max-date staleness. Latest capture 2026-08-07, staleness 0 → OK. An interior hole is structurally invisible to it.thales capture-qajudges row counts against a trailing median. 896 rows is a perfectly normal day. The data was real — just the wrong day's.- The workflow itself succeeded, so no
🔴alert fired.
Every instrument reported green while a moat day was lost. This is the
truth-divergence shape ops/DAILY_AUDIT.md exists to catch, and it was only
findable from the repo side.
Fixed here (observability tier)
Interior-gap detection in the capture ledger (capture_gaps), reported by
thales captures as status GAPPED with the missing dates listed. Scoped
to streams flagged expect_every_td (irregular cadences — trade days,
selection days, bi-monthly — would false-alarm), bounded to a 20-trading-day
window, and calendar-aware so weekends and holidays are not gaps.
Run against live data it immediately reproduced this incident and independently re-found the 2026-07-16/17 outage (the rotted-digest-test fail-close, RESEARCH.md 2026-07-17) without being told about it.
Because a detector that permanently re-reports closed incidents becomes
noise — the mechanism that cost this system seven silent weeks — gaps already
investigated are listed in ACKNOWLEDGED_GAPS with a citation to where the
explanation lives, and print as dim recorded notches rather than red alarms.
An acknowledgement without a reason string fails the tests. The 2026-08-06
skew gap is deliberately left unacknowledged until the owner rules on it.
NOT fixed here — needs the owner
The root cause is the date derivation, and changing it alters the contract of a pre-registered capture mid-accrual (the B4 activation checkpoint reads this stream in 2027-06). That is outside the audit's rails, so it is an escalation, not an edit:
-
Derive the capture date from the market session, not UTC
date.today(). The natural choice is the most recent NYSE session that has closed as of the run instant. This makes the label independent of queue delay. A cheaper partial guard: refuse to write when the target date's file already exists and the run is outside its intended window. -
Decide what to do with the 891 mislabelled rows dated 2026-08-07. Options: relabel to 2026-08-06 (recovers the lost day, but asserts a provenance the file does not record), quarantine them (honest, leaves two holes), or leave and document. Relabelling is defensible only if the 00:42Z quotes are accepted as Thursday's close — they are, but the
dtecolumn was computed against Friday and would have to be recomputed. -
data/shortability/2026-08-07.parquetholds Thursday's borrow flags. The stream is a day-over-day flip panel; a day of wrong-dated flags fabricates one non-flip and hides one real one.
Residual
The exposure window is structural: a 19:00Z cron needs only ~5h of delay to cross UTC midnight, and GitHub's shared-runner queue routinely adds ~2h. This will recur on any day the queue is bad — the outage was the trigger, not the cause.