Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

SKW-4 — the skew store serves 2,068 rows stamped on non-sessions (two NYSE holidays and a Saturday) through a reader with no trading-day filter, and per-name coverage is unmeasured: GHC and NVR have zero rows in 62 capture days and no instrument would say so

status: open · raised 2026-09-05 (external design review r2, finding 4 + a near-miss; maintainer-reproduced exactly against get_trading_days) · class: moat integrity (G4 capture layer, consumer side — the twin of GAP-1's missing days: days that should not exist) · judgement: YES for the reader filter and the coverage census (mechanical); the store disposition is the owner's standing STAMP-1 ruling · effort ~0.25 pd + the ruling · horizon: manifests on the first study that groups the stream by date; the per-name holes are permanent and grow by one notch per name per session


Plain-language summary for an owner reading one paragraph. STAMP-1 fixed the writer: the skew capture now refuses to run on a day the market is closed. But 2,068 rows written before that fix — on Juneteenth, on the observed Independence Day holiday, and on a Saturday — are still in the canonical file, and the function every study uses to read the file hands them back unfiltered. To any analysis that groups by date they look like three extra trading sessions carrying the previous session's stale quotes, inside the dataset this project calls its moat and publishes. Separately, the capture's only per-day quality check is "did roughly the usual number of rows arrive"; a name that never arrives is 0.1% of that count and invisible. Two names in the universe, GHC and NVR, have never contributed a row in 62 capture days, and they are not the two class-share names SKW-2 already covers. The fix has three parts: filter on read (one line), decide what to do with the 2,068 rows (the owner's already-open STAMP-1 ruling — quarantine recommended), and add a per-name coverage census to capture-QA.

Mechanism — verified at HEAD cd571b2, data at 3779020 (50,438 rows, 62 dates, 0 duplicate (date, symbol) keys)

Leg 1 — non-session rows served.

daterowswhy it is not a session
2026-06-19892NYSE holiday (Juneteenth) — the cron is Mon–Fri, not calendar-aware
2026-07-03283NYSE holiday (Independence Day observed)
2026-08-29893Saturday — the STAMP-1 incident (a 02:01Z Saturday run stamped the runner's UTC date)

Re-derived with thales.utils.calendar.get_trading_days; total 2,068 of 50,438. The writer was fixed by STAMP-1 (cli.py:4592-4598 refuses non-trading days and stamps the market session). load_skew (options_skew.py:316-344) returns every row verbatim; nothing on the read path re-checks. Because the (date, symbol) keys are unique, these are not double-captures — they are phantom sessions: Alpaca serves the previous close's chain on a closed day, so the rows carry drifted stale quotes under a date on which nothing was observed. This is the classic fix-sweep failure the reviewer named: the fix landed where the bug was reported (the writer), not everywhere it lives (the stored rows, every reader that trusts them).

Leg 2 — per-name coverage unmeasured.

symbolcapture days present (of 62)
GHC0
NVR0
MASI1
ASGN18
HOLX18
PSTG19
BK32
CTRA35

capture_qa's per-day skew test is total row count against a trailing median (captures.py:610-629); one permanently absent name is ~0.1% of that count. SKW-2 covers exactly two names (BRK-B, BF-B) for a known cause (dash symbology). GHC and NVR are dash-free and outside its scope, and their cause is unknown — chain absent at the venue, fetch failing, or no expiry in the [20, 45]-DTE window every day (the SKW-3 None-skew shape). Determine, don't guess: the census is the instrument; the cause is its first output.

Fix shape (propose-only)

  1. Reader filter. load_skew(..., sessions_only=True) as the default: drop rows whose date fails is_trading_day, log the count dropped. Apply the same to the term and chains loaders if they share the path. One line plus a log line; safe regardless of the ruling in 2.
  2. Store disposition — the owner's ruling, not the panel's. The STAMP-1 built-row footer records the "relabel vs quarantine" ruling for the already-mislabelled captures (08-07, 08-28, 08-29) as still open since the 08-06/07 memo. These three JSONL dates are the same ruling. Recommendation: quarantine, not relabel — the rows carry quotes that drifted from a session nobody observed, so relabelling would assert an observation that did not happen. Mechanism: a thales captures quarantine --non-session command that moves the rows to a sibling *_quarantine.jsonl with a CAPTURES.md note — never a text edit of committed data (this shop does not hand-edit history; 2026-08-29 memo).
  3. Coverage census in capture_qa: names with zero rows over the trailing N sessions, listed by name, plus a per-name presence ratio in the report. Either re-scope SKW-2 from "two dash-class names" to this census, or keep SKW-2 as the symbology fix and let this row own the census — triage's call; the two must not both claim it.

Test design + negative control (CQA-1 doctrine)

  • load_skew over a fixture containing a Saturday-dated row → excluded by default, included with sessions_only=False; revert → red.
  • capture_qa over a fixture where one name is absent for 5 sessions → named in the report; revert → silent (today's behaviour).
  • Quarantine command over a fixture with one holiday date → rows moved, key count conserved across the two files, CAPTURES.md note asserted.

Kill criterion — pre-registered

If the owner rules that non-session rows are wanted as "last-known-surface" observations, leg 1 becomes a session: bool column rather than a filter and leg 2 becomes "keep, flagged"; the row records the ruling and closes on the column. Leg 3 has no kill — an unmeasured coverage is a defect by the charter's own G4 wording regardless of what the census finds.

Cap note for triage

Fold partner SKW-2 (leg 3) / the STAMP-1 ruling (leg 2). Never GAP-1 — it owns missing days and the G4 gauge claim; this row is the mirror axis (phantom days, absent names).


Built — recorded 2026-09-06; legs 1–2 shipped, leg 3 re-routed

Leg 1 — reader filter (code PR #152). load_skew(sessions_only=True) is the default: rows stamped on non-session dates are dropped on read and the count is logged.

Leg 2 — the store, under the owner's ruling (2026-09-06). The ruling turned out to be TWO rulings, because the 2,068 rows are two classes — see the verification memo and the CAPTURES.md "Data-disposition rulings" entry written by thales capture-resession --apply:

  • Holiday re-serves (06-19: 892 rows; 07-03: 283; plus their chains files): no session was observed and the previous session already had its own rows → quarantined to data/quarantine/, never deleted.
  • STAMP-class mislabels (the 08-29-stamped 893 rows, and — found while verifying — the 08-07- and 08-28-stamped ones): genuine post-close observations of the previous session, stamped with the runner's UTC date → re-sessioned per row by captured_at (08-07→08-06 for the 00:43Z firing, 08-28→08-27, 08-29→08-28; the 5 names genuinely captured in-session on 08-07 stay), across options_skew, options_chains, options_iv_term and shortability. Two true holes remain and are acknowledged in captures.ACKNOWLEDGED_GAPS: shortability 08-07 and 08-28, whose genuine states were skipped by the writer's file-idempotency. pit_snapshots deliberately untouched (PIT-1; different date semantics).

Leg 3 — per-name coverage census: NOT built here. Re-routed to SKW-2's re-scope (the row's own either/or): GHC/NVR at 0/62 capture days and the low-coverage names are its census's first output; the mechanism stays "determine, don't guess".

Route — recorded honestly: built on the owner's direct instruction in the 2026-09-06 interactive session, NOT via queue/approved/; the guard denies the open→built rename, so a human merges.