Thales
← research journal
Aug 7, 2026raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

2026-08-07 — the skew capture lost a day and gained a contaminated one

Class: INCIDENT (data integrity, forward capture). Found by: the daily audit, from the ground-truth side. Nothing alarmed.

What happened

The 2026-08-06 runner-loss outage (audit PR #90) displaced the Skew Snapshot job. Its cron is 0 19 * * 1-5; the run was created at 2026-08-07T00:42Z, about 5.7 hours late — past UTC midnight.

thales snapshot-skew stamps its output with the runner's UTC calendar date (cli.py: snapshot_skew(..., today=_dt.date.today())). So the run that was supposed to capture Thursday filed everything under Friday:

stream2026-08-062026-08-07
options_skewabsent891 rows from the displaced run + 5 genuine
options_chainsabsentoverwritten by Friday's run (genuine)
options_iv_termabsentoverwritten by Friday's run (genuine)
shortabilityabsentdisplaced run's file, never refreshed

Two distinct failure shapes, from the same cause:

  1. A permanent hole at 2026-08-06. Free point-in-time IV does not exist retroactively — this is the failure mode the whole stream is classified TIME-GATED to prevent.

  2. Contamination of 2026-08-07, which is worse than a hole because it is invisible. snapshot_skew builds a done set of symbols already recorded for today and skips them (a resume guard for partial failures). Friday's genuine run therefore found 891 symbols "already done" and appended only the 5 that were missing. Verified: all 891 rows are byte-identical to the displaced run's write, and their dte values are computed against 2026-08-07, so they carry Thursday's quotes under Friday's label with Friday's arithmetic.

options_chains and options_iv_term write whole-file per day, so Friday's run overwrote them — those two are genuinely Friday's. shortability was not rewritten at all. The four streams from one run now disagree about what day they hold.

Why nothing fired

  • thales captures judges max-date staleness. Latest capture 2026-08-07, staleness 0 → OK. An interior hole is structurally invisible to it.
  • thales capture-qa judges row counts against a trailing median. 896 rows is a perfectly normal day. The data was real — just the wrong day's.
  • The workflow itself succeeded, so no 🔴 alert fired.

Every instrument reported green while a moat day was lost. This is the truth-divergence shape ops/DAILY_AUDIT.md exists to catch, and it was only findable from the repo side.

Fixed here (observability tier)

Interior-gap detection in the capture ledger (capture_gaps), reported by thales captures as status GAPPED with the missing dates listed. Scoped to streams flagged expect_every_td (irregular cadences — trade days, selection days, bi-monthly — would false-alarm), bounded to a 20-trading-day window, and calendar-aware so weekends and holidays are not gaps.

Run against live data it immediately reproduced this incident and independently re-found the 2026-07-16/17 outage (the rotted-digest-test fail-close, RESEARCH.md 2026-07-17) without being told about it.

Because a detector that permanently re-reports closed incidents becomes noise — the mechanism that cost this system seven silent weeks — gaps already investigated are listed in ACKNOWLEDGED_GAPS with a citation to where the explanation lives, and print as dim recorded notches rather than red alarms. An acknowledgement without a reason string fails the tests. The 2026-08-06 skew gap is deliberately left unacknowledged until the owner rules on it.

NOT fixed here — needs the owner

The root cause is the date derivation, and changing it alters the contract of a pre-registered capture mid-accrual (the B4 activation checkpoint reads this stream in 2027-06). That is outside the audit's rails, so it is an escalation, not an edit:

  1. Derive the capture date from the market session, not UTC date.today(). The natural choice is the most recent NYSE session that has closed as of the run instant. This makes the label independent of queue delay. A cheaper partial guard: refuse to write when the target date's file already exists and the run is outside its intended window.

  2. Decide what to do with the 891 mislabelled rows dated 2026-08-07. Options: relabel to 2026-08-06 (recovers the lost day, but asserts a provenance the file does not record), quarantine them (honest, leaves two holes), or leave and document. Relabelling is defensible only if the 00:42Z quotes are accepted as Thursday's close — they are, but the dte column was computed against Friday and would have to be recomputed.

  3. data/shortability/2026-08-07.parquet holds Thursday's borrow flags. The stream is a day-over-day flip panel; a day of wrong-dated flags fabricates one non-flip and hides one real one.

Residual

The exposure window is structural: a 19:00Z cron needs only ~5h of delay to cross UTC midnight, and GitHub's shared-runner queue routinely adds ~2h. This will recur on any day the queue is bad — the outage was the trigger, not the cause.