Thales
← research journal
raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

RSN-1 — capture-resession's counters (the safety check of a destructive rewrite) mis-report in three reachable cases, and the schema pin can silently drop captured_at from an owned capture file

status: open · raised panel run 21 (2026-09-11, change-review catch on PR #152 — does NOT count toward the frontier cadence) · class: moat / instrument calibration (the guard that guards the crown-jewel data) · effort ~0.5 pd · horizon: before the next holiday-class ruling — Thanksgiving 2026-11-26 is a hard date (the backup GH cron fires Mon–Fri regardless of NYSE holidays; 06-19 and 07-03 already produced this class)

Plain-language summary for an owner reading one paragraph. The 2026-09-06 STAMP-1 ruling was executed by thales capture-resession, whose apply-report counters ARE its verification step — the dropped counter is what caught the first apply's real 892-row bug. This run's change review verified the executed ruling clean (ledger totals sum exactly; dry-run at HEAD says "nothing to do") — but the counters themselves are wrong in three cases the NEXT ruling can reach, and one of them is not a counter bug but silent data loss: with a pre-2026-09-02 legacy file as the first source, the rewritten target file silently loses the captured_at column — the column the module's own docstring calls "the truth" — from an un-backfillable owned forward-capture file. A guard that lies during the next destructive apply is the cardinal-rule class.

Mechanism — ground truth verified this run at HEAD fa55f0d

All in src/thales/data/resession.py:

  1. Quarantine double-count (:322-325) — when a quarantine parquet already exists for the date, the incoming part is concat-merged with it and report["quarantined"] += part.height re-counts the pre-existing quarantined rows on every apply. Fires on any future ruling touching an already-quarantined date; inflates the CAPTURES.md ledger totals.
  2. dropped conflation + mis-attribution (:333-344) — dropped = base.height + new.height − merged.height counts pre-existing intra-base duplicates as collision drops, and under keep="first" an existing row can silently lose to its own twin (contradicting "existing rows win"); :342-344 then stamps the identical aggregate onto every move sharing the target, so a two-source→one-target ruling double-reports in the ledger.
  3. Schema pinned to sources[0]'s RAW columns (:339 with :286, :297-298) — the rewritten target is merged.select(src_frames[sources[0]].columns), and src_frames holds the raw read; the captured_at backfill mutates only the local df. A legacy source without raw captured_at → the column is silently dropped from the rewritten target. Converse: a sources[0]-only column raises ColumnNotFound mid-apply, and apply is only per-file atomic. The legacy-file scenario is exactly the scenario this tool exists for.

The 2026-09-06 ruling escaped all three only because it had no pre-existing quarantine files, no intra-base dupes, single-source targets, and homogeneous schemas.

Fix shape (propose-only)

Count only rows moved this apply into quarantined; compute per-move collision counts against base keys and report intra-base dedup separately (or refuse a base with internal duplicates); write the union schema (or an explicit per-stream stable schema) instead of sources[0]'s raw columns.

Pre-registered kill criterion / done-when

Three regression tests, each red today:

  1. Pre-existing quarantine file → quarantined equals rows moved this apply only.
  2. Two moves into one target get their own collision counts, and a base with an internal duplicate reports zero collisions.
  3. Legacy sources[0] without raw captured_at merging into a target that has it → captured_at values survive in the rewritten file (asserting presence alone does not satisfy), and a sources[0]-only column does not raise.

Idempotency test re-green after. Renaming counters without the per-case semantics does not satisfy this row.

Triage note — 2026-09-14 (annotation only; the body above is untouched)

Split recommended: leg 3 is not a counter bug and should not wait for the Thanksgiving run-up with the other two. Legs 1 and 2 mis-report a count; leg 3 (resession.py:339, merged.select(src_frames[sources[0]].columns) against the raw read at :286, while the captured_at backfill at :297-298 mutates only the local df) silently drops a column from an owned capture file that cannot be re-fetched. That is data loss, not a reporting defect, and it is ~0.15 pd of the row's 0.5 pd.

All three legs re-verified STILL PRESENT at HEAD 340bc5f: :322-325 (quarantine double-count via pl.concat before +=), :331-344 (dropped computed as an aggregate under keep="first", then stamped onto every move sharing the target), :339 (schema pinned to the raw source columns). resession.py has exactly one commit — 877dce6 (#152) — and is untouched since this row was filed.

Deadline verified, not passed. 2026-11-26 is the fourth Thursday of November; 73 days out. The recurrence claim holds: captures.py:323 records that the 06-19 and 07-03 holiday re-serves were quarantined, and .github/workflows/skew-snapshot.yml carries no holiday guard, so the backup GitHub cron will fire on Thanksgiving exactly as it did on those two.

Nothing here needs an owner ruling — it is code plus three regression tests.

DISMISSED — 2026-10-05, triage (12-cap overflow, ranked 14 of 14)

Killed because the mechanism behind its only hard date no longer exists, so nothing in it is dated any more. Reopens on any of the four conditions below. Body untouched; this is a move, not a deletion.

All three code defects are REAL and still present at HEAD 502690b — this dismissal is about timing, not validity: leg 1 src/thales/data/resession.py:322-325, leg 2 :332-344, leg 3 :339 against the raw read at :286 while the captured_at backfill at :297-298 mutates only the local frame (silent schema loss, not a counter bug).

WHY IT LOSES THE SLOT — the premise behind its horizon is falsified. The horizon reads "before the next holiday-class ruling — Thanksgiving 2026-11-26 is a hard date", and the 09-14 annotation argues that skew-snapshot.yml "carries no holiday guard, so the backup GitHub cron will fire on Thanksgiving exactly as it did on 06-19 and 07-03". True of the workflow, false of the effect: src/thales/cli.py:4656-4663 — the STAMP-1 guard, built 2026-09-02, nine days BEFORE this row was raised — computes market_session_date() and returns early if not is_trading_day(session) before snapshot_skew is ever called. The Thanksgiving cron will fire, print "not a trading day", and capture nothing. The 06-19/07-03 holiday re-serve class that would generate the next ruling cannot recur through the skew writer. (Shortability was already guarded, same comment at :4658. The second dispatcher, scripts/win/session.py:293-297, has no holiday guard of its own but invokes the same guarded CLI.)

With 2026-11-26 no longer mechanism-backed the row has no date at all: it guards a guard whose trigger is a future destructive ruling nobody has scheduled. Leg 1's blast radius is also narrower than stated — exactly two pre-existing quarantine parquets exist (data/quarantine/options_chains/2026-06-19.parquet and .../2026-07-03.parquet), so the double-count fires only on a future ruling that quarantines into those two dates in options_chains.

Against the other thirteen rows it ranks last: four are live-path safety or permanent data loss, five carry real dates inside the December window, and OSR-1/PLB-1 are cheaper and unblocked. This is the only row of the fourteen whose stated deadline dissolved under verification.

REOPEN CONDITIONS — any one suffices, no new argument needed:

  1. A destructive capture rewrite is proposed or scheduled — any ruling that would run thales capture-resession again. The counters ARE that tool's verification step (the dropped counter caught the first apply's real 892-row bug), so this must be built before the next apply, not after.
  2. A holiday-class displaced-capture incident recurs by any route — the cli.py:4659-4663 guard is bypassed, removed, or proves incomplete. Note it protects the writer; a file arriving by any other path is not covered.
  3. Leg 3 alone, at any time, on its own merits — ~0.15 pd, and it is silent data loss from an un-refetchable capture file; its trigger is any displaced-instant ruling, not only holidays. The 09-14 note already recommended splitting it out and that survives intact. If only one thing from this row is ever built, build this.
  4. A third party reads the counters as truth — any check, digest, test or routine that starts treating the apply-report as verification evidence.

Not reopened by: 2026-11-26 arriving. That is the premise shown false above.