Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

PIT-1 — the pit snapshotter greens while producing nothing (or garbage that permanently occupies its slot): three silent shapes, bleeding under the displaced-cron regime

status: open · raised panel run 16 (2026-09-01), executing the 2026-08-31 daily-audit memo's detector hand-off (item c), change-review verified and adversarially narrowed · class: moat / loudness (capture path — the P0 shape: a green step with an empty product) · judgement: YES · effort ~0.5 pd · horizon: live now — bleeds on any momentum run displaced past Tiingo's EOD publication; two of the last four trading days already bled permanently


Plain-language summary for an owner reading one paragraph. The point-in-time signal snapshotter — one of the forward-capture streams the charter calls irreplaceable — can finish green while writing nothing, and can write a nearly-empty file that then blocks its own repair forever. Last week's displaced crons exercised all of it: 08-26 is missing outright, 08-27 and 08-28 hold 123 and 58 rows against a ~890 median and can never be healed (write-once), and 08-31's run wrote nothing at all behind a green "Point-in-time snapshot" step. The proposed fix makes the writer loud and self-protecting: refuse thin writes (so the slot stays open for a same-day retry), and make "nothing was written" a visibly non-green outcome — without ever letting a capture failure fail-close trading.

Mechanism — three shapes, all verified at HEAD 29c6f23

  1. Exit-1 swallowed green. The no-data paths exit 1 (src/thales/cli.py:4700-4702, :4725-4727) but the workflow step is continue-on-error: true (.github/workflows/paper-trading.yml, deliberately, so a capture failure can never fail-close trading — that design choice is correct and this row must preserve it), so the job shows green with nothing produced.
  2. Stale-date idempotent skip. cli.py:4706 defaults the snapshot date to prices["date"].max(). A displaced run that fires before Tiingo publishes today's EOD sees yesterday as max; yesterday's snapshot already exists → write-once skip (src/thales/data/pit_snapshot.py:106-108) → exit 0, honestly green, nothing written. This is what happened 08-31 (run commit bca9ae3 contains no pit file).
  3. Thin write permanently occupies the slot. build_snapshot has no row floor, so a very-late run persists whatever partial EOD rows the vendor has published (08-27 = 123 rows, 08-28 = 58 vs ~890 median), and write-once idempotency then blocks any heal — pit has no heal path (skew has one, options_skew.py:433-448).

Honesty note: the thin writes were not fully silent — capture-qa's existing 50%-of-trailing-median floor caught 08-28, which is the repeat 🔴 mail. The true claim is: green workflow, permanent slot consumption, and misattributed alerting (the alert-naming half is CQA-2's row).

Boundary vs the STAMP family (state it so triage doesn't merge them): STAMP-1/2 are defects of date.today() — the runner's UTC wall clock used as stamp and idempotency key. PIT-1's date comes from prices["date"].max(), a data-lag-derived date; the STAMP fixes (ET normalization, trading-day guard) never touch cli.py:4706 and would not stop this bleed. Distinct mechanism, distinct fix site. GAP-1's capture_gaps detects interior holes after the fact and deliberately stops at the stream's own latest capture, so it cannot prevent (or even promptly see) a trailing hole; SAC-1 is trading-path persist sites. No overlap.

Fix shape (propose-only; narrowed by adversarial review)

  1. Row floor at the writer: refuse to persist a snapshot below the SAME 50%-of-trailing-median convention capture_qa already commits to for this stream (src/thales/data/captures.py:819-826) — reuse that function/threshold, introduce no second tunable number. A refused thin write leaves the slot open for a later-same-day retry (heal-lite for free).
  2. "Nothing written" becomes a distinct non-green outcome surfaced on the observability channel (a separate failing step after the trading steps, or a QA/digest condition naming pit_snapshots) — while the trading job still completes.
  3. A heal path modeled on skew'sstruck by review as a second mechanism/new surface; parked as a separate owner decision if the retry window proves insufficient in practice.

Test design + kill criterion (pre-registered, negative-controlled)

  • No-quote fixture → the observability channel reds naming pit_snapshots, AND a co-registered negative control asserts the trading job still completes on the same fixture. (Without that clause the fix's laziest implementation — deleting continue-on-error — passes its own test while violating the never-fail-close-trading doctrine.)
  • Thin fixture (rows < the reused floor) → write refused loudly, file absent, slot still writable; revert the floor → only its test reds.
  • Stale-date fixture (prices max < today on a trading day) → distinct "nothing written for <today>" outcome, not a silent green skip.
  • Falsifier: if the reused floor rejects a legitimately small day (real universe shrink) within 20 shadow trading days, the floor convention is wrong — record it and re-derive from history; do not tune the threshold after seeing the failure. Since the floor is capture-qa's own, such a failure indicts a convention the QA layer already committed to, which is the honest outcome either way.

Data plan

None — owned committed state (data/pit_snapshots/, workflow logs, git history) plus static code. No external source, no cost.

Urgency, stated honestly

The 08-31 hole was healable only if the 09-01 run fired before ~23:30Z; the panel is propose-only and cannot act on that. Under Monday's ~5.7h delay mode the defect is a benign one-day lag; under the ~9h mode it bleeds a permanent day each session. Which mode this week runs in is not knowable from here — the row's case is that the writer should not depend on the scheduler's mood.