Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

RSN-1 — capture-resession's counters (the safety check of a destructive rewrite) mis-report in three reachable cases, and the schema pin can silently drop captured_at from an owned capture file

status: open · raised panel run 21 (2026-09-11, change-review catch on PR #152 — does NOT count toward the frontier cadence) · class: moat / instrument calibration (the guard that guards the crown-jewel data) · effort ~0.5 pd · horizon: before the next holiday-class ruling — Thanksgiving 2026-11-26 is a hard date (the backup GH cron fires Mon–Fri regardless of NYSE holidays; 06-19 and 07-03 already produced this class)


Plain-language summary for an owner reading one paragraph. The 2026-09-06 STAMP-1 ruling was executed by thales capture-resession, whose apply-report counters ARE its verification step — the dropped counter is what caught the first apply's real 892-row bug. This run's change review verified the executed ruling clean (ledger totals sum exactly; dry-run at HEAD says "nothing to do") — but the counters themselves are wrong in three cases the NEXT ruling can reach, and one of them is not a counter bug but silent data loss: with a pre-2026-09-02 legacy file as the first source, the rewritten target file silently loses the captured_at column — the column the module's own docstring calls "the truth" — from an un-backfillable owned forward-capture file. A guard that lies during the next destructive apply is the cardinal-rule class.

Mechanism — ground truth verified this run at HEAD fa55f0d

All in src/thales/data/resession.py:

  1. Quarantine double-count (:322-325) — when a quarantine parquet already exists for the date, the incoming part is concat-merged with it and report["quarantined"] += part.height re-counts the pre-existing quarantined rows on every apply. Fires on any future ruling touching an already-quarantined date; inflates the CAPTURES.md ledger totals.
  2. dropped conflation + mis-attribution (:333-344) — dropped = base.height + new.height − merged.height counts pre-existing intra-base duplicates as collision drops, and under keep="first" an existing row can silently lose to its own twin (contradicting "existing rows win"); :342-344 then stamps the identical aggregate onto every move sharing the target, so a two-source→one-target ruling double-reports in the ledger.
  3. Schema pinned to sources[0]'s RAW columns (:339 with :286, :297-298) — the rewritten target is merged.select(src_frames[sources[0]].columns), and src_frames holds the raw read; the captured_at backfill mutates only the local df. A legacy source without raw captured_at → the column is silently dropped from the rewritten target. Converse: a sources[0]-only column raises ColumnNotFound mid-apply, and apply is only per-file atomic. The legacy-file scenario is exactly the scenario this tool exists for.

The 2026-09-06 ruling escaped all three only because it had no pre-existing quarantine files, no intra-base dupes, single-source targets, and homogeneous schemas.

Fix shape (propose-only)

Count only rows moved this apply into quarantined; compute per-move collision counts against base keys and report intra-base dedup separately (or refuse a base with internal duplicates); write the union schema (or an explicit per-stream stable schema) instead of sources[0]'s raw columns.

Pre-registered kill criterion / done-when

Three regression tests, each red today:

  1. Pre-existing quarantine file → quarantined equals rows moved this apply only.
  2. Two moves into one target get their own collision counts, and a base with an internal duplicate reports zero collisions.
  3. Legacy sources[0] without raw captured_at merging into a target that has it → captured_at values survive in the rewritten file (asserting presence alone does not satisfy), and a sources[0]-only column does not raise.

Idempotency test re-green after. Renaming counters without the per-case semantics does not satisfy this row.