Thales
← research journal
raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

PIT-1 — the pit snapshotter greens while producing nothing (or garbage that permanently occupies its slot): three silent shapes, bleeding under the displaced-cron regime

status: open · raised panel run 16 (2026-09-01), executing the 2026-08-31 daily-audit memo's detector hand-off (item c), change-review verified and adversarially narrowed · class: moat / loudness (capture path — the P0 shape: a green step with an empty product) · judgement: YES · effort ~0.5 pd · horizon: live now — bleeds on any momentum run displaced past Tiingo's EOD publication; two of the last four trading days already bled permanently

Plain-language summary for an owner reading one paragraph. The point-in-time signal snapshotter — one of the forward-capture streams the charter calls irreplaceable — can finish green while writing nothing, and can write a nearly-empty file that then blocks its own repair forever. Last week's displaced crons exercised all of it: 08-26 is missing outright, 08-27 and 08-28 hold 123 and 58 rows against a ~890 median and can never be healed (write-once), and 08-31's run wrote nothing at all behind a green "Point-in-time snapshot" step. The proposed fix makes the writer loud and self-protecting: refuse thin writes (so the slot stays open for a same-day retry), and make "nothing was written" a visibly non-green outcome — without ever letting a capture failure fail-close trading.

Mechanism — three shapes, all verified at HEAD 29c6f23

  1. Exit-1 swallowed green. The no-data paths exit 1 (src/thales/cli.py:4700-4702, :4725-4727) but the workflow step is continue-on-error: true (.github/workflows/paper-trading.yml, deliberately, so a capture failure can never fail-close trading — that design choice is correct and this row must preserve it), so the job shows green with nothing produced.
  2. Stale-date idempotent skip. cli.py:4706 defaults the snapshot date to prices["date"].max(). A displaced run that fires before Tiingo publishes today's EOD sees yesterday as max; yesterday's snapshot already exists → write-once skip (src/thales/data/pit_snapshot.py:106-108) → exit 0, honestly green, nothing written. This is what happened 08-31 (run commit bca9ae3 contains no pit file).
  3. Thin write permanently occupies the slot. build_snapshot has no row floor, so a very-late run persists whatever partial EOD rows the vendor has published (08-27 = 123 rows, 08-28 = 58 vs ~890 median), and write-once idempotency then blocks any heal — pit has no heal path (skew has one, options_skew.py:433-448).

Honesty note: the thin writes were not fully silent — capture-qa's existing 50%-of-trailing-median floor caught 08-28, which is the repeat 🔴 mail. The true claim is: green workflow, permanent slot consumption, and misattributed alerting (the alert-naming half is CQA-2's row).

Boundary vs the STAMP family (state it so triage doesn't merge them): STAMP-1/2 are defects of date.today() — the runner's UTC wall clock used as stamp and idempotency key. PIT-1's date comes from prices["date"].max(), a data-lag-derived date; the STAMP fixes (ET normalization, trading-day guard) never touch cli.py:4706 and would not stop this bleed. Distinct mechanism, distinct fix site. GAP-1's capture_gaps detects interior holes after the fact and deliberately stops at the stream's own latest capture, so it cannot prevent (or even promptly see) a trailing hole; SAC-1 is trading-path persist sites. No overlap.

Fix shape (propose-only; narrowed by adversarial review)

  1. Row floor at the writer: refuse to persist a snapshot below the SAME 50%-of-trailing-median convention capture_qa already commits to for this stream (src/thales/data/captures.py:819-826) — reuse that function/threshold, introduce no second tunable number. A refused thin write leaves the slot open for a later-same-day retry (heal-lite for free).
  2. "Nothing written" becomes a distinct non-green outcome surfaced on the observability channel (a separate failing step after the trading steps, or a QA/digest condition naming pit_snapshots) — while the trading job still completes.
  3. A heal path modeled on skew's — struck by review as a second mechanism/new surface; parked as a separate owner decision if the retry window proves insufficient in practice.

Test design + kill criterion (pre-registered, negative-controlled)

  • No-quote fixture → the observability channel reds naming pit_snapshots, AND a co-registered negative control asserts the trading job still completes on the same fixture. (Without that clause the fix's laziest implementation — deleting continue-on-error — passes its own test while violating the never-fail-close-trading doctrine.)
  • Thin fixture (rows < the reused floor) → write refused loudly, file absent, slot still writable; revert the floor → only its test reds.
  • Stale-date fixture (prices max < today on a trading day) → distinct "nothing written for <today>" outcome, not a silent green skip.
  • Falsifier: if the reused floor rejects a legitimately small day (real universe shrink) within 20 shadow trading days, the floor convention is wrong — record it and re-derive from history; do not tune the threshold after seeing the failure. Since the floor is capture-qa's own, such a failure indicts a convention the QA layer already committed to, which is the honest outcome either way.

Data plan

None — owned committed state (data/pit_snapshots/, workflow logs, git history) plus static code. No external source, no cost.

Urgency, stated honestly

The 08-31 hole was healable only if the 09-01 run fired before ~23:30Z; the panel is propose-only and cannot act on that. Under Monday's ~5.7h delay mode the defect is a benign one-day lag; under the ~9h mode it bleeds a permanent day each session. Which mode this week runs in is not knowable from here — the row's case is that the writer should not depend on the scheduler's mood.

Triage note — 2026-09-14 (annotation only; the body above is untouched)

Honest de-escalation, recorded because it cuts against keeping this row high: no new bleed since it was raised. data/pit_snapshots/ runs 15.1–15.6 KB for every session 09-01 → 09-10, with no gaps beyond weekends and the 09-07 Labor Day holiday. The body's one named live harm has healed — it said the 08-31 hole was recoverable only if the 09-01 run fired before ~23:30Z, and data/pit_snapshots/2026-08-31.parquet exists at 15,650 B.

The proximate cause was cron displacement, and ops/launchd/ com.thales.dispatch-trading.plist (2026-09-02) now dispatches at local time, so the ~9h-delay mode is rarer. A sleeping Mac or a GitHub queue delay still reproduces it, and the body's own argument — the writer should not depend on the scheduler's mood — survives that.

The mechanism is untouched at HEAD 340bc5f. Shape 2: cli.py:4958 still snap_date = date_str or str(prices["date"].max()), and pit_snapshot.py:106-108 still if path.exists() and not force: return None. Shape 3: build_snapshot (pit_snapshot.py:34) has no row floor; cli.py:4977-4979 exits 1 only on a fully empty frame. Shape 1: paper-trading.yml:180 keeps continue-on-error: true on the snapshot step (correctly).

Partial closure not recorded in the body: paper-trading.yml:207-226 now carries a PiT freshness heartbeat in a step that is NOT continue-on-error, redding at PIT_AGE > 5 days, fixed 2026-09-02 to read the filename date rather than mtime. That catches a stopped snapshotter — not a single lost day and not a thin write, which are the two shapes this row is about.

Damage already on the books, unchanged: 2026-08-26 absent; 2026-08-27 at 4,804 B and 2026-08-28 at 3,807 B against a ~15.3 KB median — both permanently thin, and both are why the 2026-09-09 decision-grade placebo null rested on ~56 usable panel dates.

TRIAGE ANNOTATION 2026-09-21 — de-escalated a second week running; no new bleed

data/pit_snapshots/ is unbroken through 2026-09-17 (the 09-18 run writes the 09-17 date by design). No new thin or missing day since this row was raised on 09-01. The mechanism is entirely unchanged at HEAD — cli.py:4958 snap_date = date_str or str(prices["date"].max()); pit_snapshot.py:106-108 write-once skip; build_snapshot (:34) has no row floor; cli.py:4979-4981 exits 1 only on a fully empty frame; the workflow step is still continue-on-error: true, which is correct and must be preserved.

So the honest reading, third time of asking: the damage already on the books (08-26 missing, 08-27/08-28 thin at 123 and 58 rows against a ~890 median) is permanent and unrecoverable, and the row buys protection against the NEXT displaced-cron regime, not repair of the last one. It inherits CQA-2's missing-today detection leg as of this run's overflow dismissal; CQA-2's reopen condition (d) fires if this row leaves the queue without it.

Watch item for the next sweep: 2026-09-21 produced no trading run at all (Actions billing outage), so the 09-18-dated snapshot is the last one — check whether 09-21 is a permanent hole once Actions is restored, and whether the write-once skip blocks a same-day retry of it.

Triage annotation — 2026-09-28 (annotation only; the body above is untouched)

THE 2026-09-21 WATCH ITEM HAS FIRED — and the answer is better than feared. That annotation asked whether 2026-09-21 would become a permanent hole once Actions was restored. Measured this run: data/pit_snapshots/ holds 67 files, newest 2026-09-17.parquet (15,222 B). Missing data-dates: 2026-09-18, 09-21, 09-22, 09-23, 09-24, 09-25 — six trading days, up from zero missing at the last triage.

But the cause is the billing outage, not this row's mechanism, and the distinction decides the disposition. No run fired at all, so no thin file occupies those slots — the write-once skip (pit_snapshot.py:106-108) does not block a heal, and thales snapshot-pit takes an explicit date_str (cli.py:4958), so all six days are backfillable once data/raw is refreshed. That is categorically different from 2026-08-27/28, which are occupied by thin files and are permanent. Those remain the row's evidence and are re-measured unchanged: 08-26 absent; 08-27 = 4,804 B; 08-28 = 3,807 B against a ~15.3 KB median (next-smallest is 08-17 at 12,825 B, correctly not flagged).

One thing will be loud on restore, and it is worth expecting. The PiT freshness heartbeat (paper-trading.yml:212-226, not continue-on-error, reds at PIT_AGE > 5) sees a newest file of 2026-09-17 — 11 days old today. The first restored momentum run will red on it. That is the gate working, not a new defect; it does not cover either of the row's two shapes (a single lost day, and a thin write), which remain uncovered.

Mechanism unchanged at HEAD 864da4e: paper-trading.yml:181 continue-on-error: true on the snapshot step (correct, must be preserved); cli.py:4958 unchanged; pit_snapshot.py:106-108 write-once skip; and build_snapshot (pit_snapshot.py:34) still has no row floor, with cli.py:4979-4981 exiting 1 only on a fully empty frame.

Blocked on: nothing technical. It still inherits CQA-2's missing-today detection leg — CQA-2's reopen condition (d) fires if PIT-1 ever leaves the queue without that leg, so a builder must carry it. Adjacent but separate: dismissed GAP-1 was re-annotated 2026-09-26 with a derived ≈2026-10-23 deadline to enter the outage days into ACKNOWLEDGED_GAPS (captures.py:295); that is GAP-1's clock, not this row's, and nothing sweeps a dismissed row.

De-escalated for a third week on the permanent-damage measure, escalated on nothing. Effort unchanged at ~0.5 pd.

Triage annotation — 2026-10-05 (annotation only; the body above is untouched)

THIS ROW'S OWN WATCH ITEM IS ANSWERED, AND THE ANSWER IS THE WORST ONE. The 09-21 annotation asked whether 2026-09-21 would become a permanent hole, and recorded "no new thin or missing day since this row was raised on 09-01." That sentence is now false. The store runs 2026-06-05 → 2026-10-02 (70 files, each run writing the previous trading day). Missing trading dates from 08-10 forward: 2026-08-26, 09-18, 09-21, 09-22, 09-23, 09-24, 09-25, 09-28, 09-29 — eight new permanent holes since the row was raised, all but one inside the 2026-09-21 → 09-30 Actions billing outage.

These holes were LOUD, not silent (Actions refused every job; nothing pretended to succeed), so they are not this row's three shapes firing. They matter here for a different reason: the equity ledgers were healed from broker truth for exactly those dates, and this stream has no heal path. The same outage that cost the account nothing cost the point-in-time signal panel eight sessions that cannot be reconstructed at any price. That is the asymmetry the charter calls irreplaceable, measured.

All three shapes still live at HEAD 502690b. Shape 2: src/thales/cli.py:4958 snap_date = date_str or str(prices["date"].max()) plus the write-once skip at src/thales/data/pit_snapshot.py:106-108. Shape 3: build_snapshot (pit_snapshot.py:34) has no row floor — grep "floor|median|min_rows" pit_snapshot.py returns zero — and cli.py:4977-4979 exits 1 only on a fully empty frame. Shape 1: .github/workflows/paper-trading.yml:221-228 still continue-on-error: true (correct, and must be preserved), with the PiT age heartbeat non-soft at :256-269. Thin-day damage re-measured from the parquet footers: 2026-08-27 = 123 rows, 2026-08-28 = 58 rows against 887 on both 2026-09-17 and 2026-10-02 — the body's "~890 median" is accurate. The reusable 50%-of- trailing-median floor has moved to src/thales/data/captures.py:832-839 (was :819-826).

NEW SURFACE — the fix brief is now bigger than the body says, and tomorrow is why. Since PR #200 the trading day also runs off GitHub: scripts/win/session.py:267 is Step("pit snapshot", th("snapshot-pit"), soft=True), which reproduces the swallow exactly, with the same >5-day heartbeat at session.py:197-204. After the tbx-win cutover (scheduled 2026-10-06 evening) that becomes the primary surface and GitHub the backup, so any fix must land in BOTH dispatchers or it will protect only the leg that no longer runs the job. A builder taking this row after the cutover and patching paper-trading.yml alone would ship a fix that never executes.

Rank this week: 4 of 12 (P0 — the only row in the tree where the loss is permanent and the clock is every trading day). BLOCKED on: nothing technical. No owner ruling, no data, no credentials — it needs the approval move and 0.5 pd. It is the clearest single answer to "what would I regret not having decided a month from now": a month of this row sitting open is ~21 more sessions of a stream whose gaps cannot be bought back.

Approved by the steward — 2026-10-08

Approved by the steward under the owner's 2026-10-07 delegation: P0 under NORTHSTAR §4 (a green step with an empty or garbage product on a forward capture whose holes are permanent); the fix shape is narrowed by adversarial review to two legs that reuse capture_qa's existing 50%-of-trailing-median threshold and introduce no new tunable number; the test design and kill criterion are pre-registered and negative-controlled; triage records it needs no owner ruling, data or credentials. Build legs 1–2 only (leg 3, the heal path, stays struck). Constraint for the build: the trading job must still complete — the new non-green outcome belongs on the observability channel, never in the pre-trade gate, and the PR must state that tomorrow's orders are unchanged.

Built — 2026-10-15 (interactive session for the owner) · outcome: shipped

Route, recorded honestly. Built in an interactive Claude Code session on the owner's instruction ("take care of what you can for me", 2026-10-15), not by the implementer: every cloud routine, the implementer included, was refused on an account usage limit from 10-12, and this row's bleed fired twice in that window (10-12 = 49 rows, 10-13 = 1 row; CAPTURES.md). The approval is the steward's of 2026-10-08, unchanged. The session that wrote this does not merge it: a later steward run reviews it cold, or the owner merges.

Leg 1, the writer floor. write_snapshot refuses a snapshot below captures.pit_row_floor: the same 50%-of-trailing-median floor capture_qa applies to this stream, now one function with two callers and no new number. A refusal writes nothing, so the slot stays open for the next morning's run; --force does not bypass it. snapshot-pit names each outcome: written, already captured (exit 0), refused thin (exit 1), slot held by a thin file (exit 1). No dispatcher changed, because both call the same command: paper-trading.yml keeps continue-on-error: true and tbx-win keeps soft=True.

Leg 2, "nothing written" is red on the observability channel. capture-qa now fails pit_snapshots, naming the date, when the previous session has no snapshot at all. This is CQA-2's missing-today leg, built here for the PiT stream per CQA-2 reopen condition (d). It runs where capture-qa already runs (the capture job on GitHub and tbx-win's skew session), never in a trading job. The lag negative control holds: the normal one-session lag, Mondays, weekends and holidays stay green. Recorded notches in ACKNOWLEDGED_GAPS stay dim.

One departure from the body, with the reason. The stale-date fixture asked that a run which writes nothing be non-green. Since the tbx-win cutover, GitHub's backup leg finds the session already captured by the box nearly every day, so a red for that case would be a daily false alarm: the class CQA-2's lag control forbids. The run now says so ("Nothing written: <date> is already captured") and stays green. The red belongs to what is actually lost: a session with no snapshot (capture-qa), or a slot held by a fragment (exit 1).

Tests: tests/test_data/test_pit_floor.py, in the observability tier, so it can never fail-close the pre-trade gate. They implement the pre-registered fixtures: thin, no-quote, stale-date, the lag control, and the co-registered trading-completes control on BOTH dispatchers. Negative controls are recorded in the PR.

Falsifier, carried: if the floor refuses a legitimately small day (a real universe shrink) within the first 20 trading days after merge, the convention is wrong: record it and re-derive it from history; never tune it after the failure.

Not done: leg 3 (the heal path) stays struck. 10-12 and 10-13 stay truncated; rewriting a write-once capture is the owner's decision.