Thales
← research journal
raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

STAMP-2 — the trading path shares STAMP-1's UTC-date defect: a midnight-crossing delayed run consumes the NEXT session's idempotency slot, the real session is silently skipped, and the no-run canary reads green

status: open · raised panel run 14 (2026-08-28) · class: loudness & safety (live path) · judgement: YES · effort ~0.5 pd (shares the ET-normalization helper STAMP-1 specifies — build together) · horizon: recurrence now measured twice in three weeks, outage-correlated

Plain-language summary for an owner reading one paragraph. GitHub's runner queue delayed all of yesterday's crons by ~9 hours. The momentum and VRP runs crossed UTC midnight and stamped their records 2026-08-28 — so when today's real crons fire during market hours, both sleeves will see "already ran today" and silently do nothing: no in-session kill-switch pass, no in-session VRP manage pass, for the second consecutive session (08-27's pass did execute — late, after the close). Every detector shows green, because the same wrong date that loses the session also fakes the record the detectors check. This is STAMP-1's defect (open since panel run 2) materializing on the trading path, which STAMP-1 explicitly did not cover.

Mechanism, code-verified. daily.py:1341-1342 and vrp_daily.py:488 default today = date.today() — the runner's UTC calendar date. The paper-trading workflows pass --skip-market-check (paper-trading.yml:172, paper-trading-vrp.yml:88), so no session anchor remains except that UTC date. state.py:180 (already_ran_today) then uses the same date as the idempotency key, and daily.py:1360 / vrp_daily.py:498 skip the run when a run_summary for it exists. A delayed run that starts before midnight UTC but persists after (momentum: started 23:53:22Z, wrote 00:05:50Z), or fires entirely after (VRP: 00:00:18Z), stamps TOMORROW — and tomorrow's real run skips.

It materialised unattended, 2026-08-27→28 (second occurrence of the class; first on the trading path).

  • momentum run_summary {"date":"2026-08-28","vol_check_only":true} written 00:05:50Z (commit 2123220) on evening-of-08-27 equity → the real 08-28 session gets no kill-switch/vol pass (skip at daily.py:1360).
  • vrp run_summary {"date":"2026-08-28","action":"manage:hold"} written 00:01:57Z (9a6615d) → the real 08-28 manage pass is skipped (vrp_daily.py:498).
  • meanrev started 23:55:55Z — still 08-27 UTC — and stamped correctly; its 55 after-hours selection orders filled at the 08-28 open (TCA damage only). The four-minute spread between meanrev and VRP is the whole difference between "correct" and "slot consumed": the margin is minutes, not hours.
  • momentum equity_history now carries {"date":"2026-08-28","equity":13687.36} — the 08-27-evening value. Friday's true close will never be recorded: the 08-27→28 return reads ~0 and Monday's row absorbs two days. This series feeds both 09-08 clocks (placebo decision-grade + oos-monitor first verdict).
  • pit_snapshots/2026-08-27.parquet: 123 rows vs trailing median 890 — capture-qa caught it (the one loud alarm of the night; operator emailed 03:38Z).
  • the skew stream repeated STAMP-1 verbatim: the 03:31Z run stamped 894 skew records / 61,478 chain rows / 8,138 term rows / 14,279 shortability names 2026-08-28 on an 08-27 after-hours basis, QA-green. The 08-27 capture day is a permanent gap, and the real 08-28 session is lost when the 19:00Z run no-ops — now 4 of ~57 accrued dates on the B4-pinned stream holed or mislabelled. Stated honestly: no pinned B4 statistic is threatened — the pin samples one signal date per ISO week and week 35 is covered by 08-26 (893 valid rows, 2.2× the 400-row floor); the residual is that the mislabelled 08-28 rows, if left unquarantined, would be mechanically selected as week 35's signal date under the pin's "last qualifying date" rule (the relabel-vs-quarantine ruling from the 08-06/07 collision memo is still pending and has until mid-2027 to land).

Why this is a new row, not STAMP-1 relitigation (C0). STAMP-1 scopes itself to the capture writers explicitly. TECH_DEBT.md:661-664 names daily.py/vrp_daily.py but records only the LABEL error, rated "MED, low practical" on a premise ("a manual/delayed run… would stamp the wrong trading day") that stops one step short: the practical harm is not the wrong stamp, it is the consumed idempotency slot — a silently skipped live session. That consequence is recorded nowhere for the trading path.

The genuinely new half: the detector is blind by construction — and last night it fired backwards. The digest's no-run canary (fleet digest --fail-on-no-run; logic at fleet_digest.py:144-145) asks "does a run record exist for this trading day?" — and the mislabelled record satisfies it. Last night the digest itself crossed midnight (started 00:37:41Z), so its today was 08-28, and the sleeve its no-run job reddened was meanrev — the one sleeve that behaved correctly (stamped 08-27, hence no 08-28 record yet), while momentum and vrp, whose real sessions are the ones lost, read green off their mislabelled 08-28 records. One run, a false positive on the healthy sleeve and false negatives on both broken ones. Tonight it reads all-green while two sleeves skip their real session. Defect and detector are coupled through the same bad date, the exact shape GAP-1 recorded for capture staleness (max(date) cannot see interior/mislabelled holes). A run-record census keyed on the trading calendar would flag the missing day; nothing keyed on "does a record exist for today" ever can.

Fix shape (propose-only, owner decides). One ET-trading-date helper — the same one STAMP-1's fix needs; build them together — plus a refuse-or-restamp guard at the two writers: a run whose wall-clock falls outside the stamped ET session (pre-open grace → close+grace) either restamps to the session it actually belongs to, or exits non-zero naming both dates. Never silently consume a future slot. (Whether a late run should still execute a catch-up vol-check for yesterday's session is a design choice the guard makes explicit instead of accidental.)

Kill criterion — pre-registered, negative-controlled in both directions. On a temp copy with the clock frozen at 2026-08-28T00:05:50Z and 08-27's state in place, the momentum writer must stamp 2026-08-27 or refuse with a non-zero exit naming the date — never write an 08-28 run_summary. Negative control: replay the stamp decision against the actual (date, start-time) pairs of the 30 scheduled paper-trading runs ending 2026-08-26 — pinned here so the 08-27→28 midnight-crossing incident runs are excluded from the must-not-fire set by construction, not by post-hoc argument — and assert the guard fires on none; replay a same-session re-run and assert the idempotent skip still holds. Then shadow 20 trading days: if the guard fires on any normally-timed run, revert it and record the miss as permanent rather than tuning the threshold after the data exists.

Effort ~0.5 pd. Horizon: not time-critical in code, but each recurrence costs a full session's risk-monitoring pass on every affected sleeve and pollutes the equity series feeding pinned clocks; recurrence is outage-correlated — it fires precisely when the system is already degraded, same as STAMP-1's.

Owner flags recorded with the row (one-time surgery, deliberately NOT part of this row's scope; all propose-only):

  1. Decide whether to correct data/processed/equity_history.jsonl's 08-28 row from broker truth before the 09-08 verdicts read the series (backfill-equity fills holes; it does not overwrite wrong values).
  2. The mislabelled 2026-08-28 skew/chains/iv_term/shortability files: STAMP-1's recorded recovery trap applies verbatim — a re-dispatch is a pure no-op, and --out data/<anything> silently overwrites; --out /tmp/... is the safe capture path if any of today's real session is to be salvaged before/after 19:00Z.
  3. fleet-digest.yml carries two contradictory comments about the digest beacon on the no-run path (~line 97-99: "deliberately … keeps the beacon pinging" vs ~line 117-118: "success() is false on the exit-2 no-run path … withholds the ping"). Observed behaviour last night: the beacon pinged. One comment is rot; the weekly audit should pick which is the design.

BUILT — 2026-09-02 (together with STAMP-1)

Outcome: shipped. DailyExecutionWrapper.run and VRPExecutionWrapper.run resolve today through market_session_date() (the session the run observes: ET date, pre-open → previous trading day), so a midnight-UTC-crossing run stamps the session it belongs to and hits the EXISTING idempotency gate for that date instead of consuming tomorrow's slot. The fleet digest's no-run canary judges the same session date (the backwards fire of 08-28 cannot recur). The design choice the row left explicit — catch-up vs refuse — is answered as RESTAMP: a late run belongs to the session it observed; if that session already ran it is the idempotent skip, if it did not it is the catch-up. The divergence from the UTC date is logged loudly either way.

Route — recorded honestly: built on the owner's direct instruction in an interactive session (2026-09-02 self-audit, finding M4), NOT via queue/approved/; a human merges.

Kill criterion — met, negative-controlled both ways (tests/test_utils/test_market_session_date.py): clock frozen at 2026-08-28T00:05:50Z with 08-27's state in place → the momentum wrapper resolves 2026-08-27 and takes the "Already ran today (2026-08-27)" skip — no 08-28 run_summary is written; the VRP instant 00:00:18Z likewise. Negative control: the stamp decision replayed against the actual creation instants of the 30 scheduled paper-trading runs ending 2026-08-26 (pinned in the test, incident runs excluded by construction) fires on none — every one stamps exactly its UTC date; a same-session re-run still takes the idempotent skip. The 20-trading-day shadow the row asks for starts at merge.

Owner flags from the row: (1) the 08-28 momentum mark was corrected from broker truth in PR #125; VRP's 08-28 mark remains the known phantom (backfillable, no clock before ~Jan 2027); (2) the mislabelled 08-28 capture files remain as recorded in CAPTURES.md pending the relabel-vs-quarantine ruling; (3) the two contradictory fleet-digest beacon comments are untouched here (weekly audit's call).

Provenance recorded by triage 2026-09-07 (annotation only; the record above is untouched): shipped in PR #132 (100cf61, merged 2026-09-03), jointly with STAMP-1. The built section above cites #125 only for an owner flag, not for the shipping merge.