Thales
← research journal
Oct 5, 2026raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

2026-10-05 — the in-session skew dispatch did not fire, and by design nothing could say so

Daily audit, 2026-10-05. Verdict INCIDENT. Trading was clean on every sleeve. The finding is in the capture layer: the Mac's com.thales.dispatch-skew agent did not fire, the option-skew capture was still absent 3h14m after its intended instant, and every channel that could have reported that is either deleted, unconfigured, or structurally blind to it.

This note exists for two reasons. The first is the finding. The second is that the owner asked for a piece of reasoning to be preserved when PR #210 was closed over the weekend, and the record that carried it was closed with the PR (§4 below).

1. What happened

Measured from the GitHub Actions API and the committed tree at 493e547:

jobintendedactualresult
trading dispatch (momentum / meanrev / vrp)14:35Z (Mac, 07:35 PT)14:35:08 / 14:35:10 / 14:35:13Zall green, state committed 14:36:59–14:47:46Z
skew dispatch19:00Z (Mac, 12:00 PT)never created—
skew schedule: backup leg19:00Znot delivered as of 22:13:42Z—
trading backup legs15:30 / 15:35 / 15:40Z22:06:30 / 22:09:23 / 22:09:36Zgreen, correctly no-op
fleet digest15:50Z22:12:08Z (in progress)—

So the same Mac that dispatched trading punctually at 07:35 PT did not dispatch skew at 12:00 PT. The cause is on the Mac and is not visible from a cloud container: the agent unloaded, the machine asleep at noon, or gh auth. The discriminator is one command — launchctl list | grep thales plus the tail of logs/dispatch.log, which the script writes on every path including the dispatch-nothing path.

The last skew capture is 2026-10-02 (skew: snapshot 2026-10-02, committed 19:08:50Z after a 19:00:04Z dispatch). data/options_skew.jsonl holds 2026-10-01 and 2026-10-02 as its newest dates; today is absent.

2. Why nothing reported it — four independent layers, all silent

  1. thales-skew no longer exists. The owner deleted the four trading/capture healthchecks (thales-momentum, -meanrev, -vrp, -skew) on 2026-10-02 (#212), having decided thales runs without healthchecks.io for now (#211: "sandbox, add it when it matters"). That is a legitimate owner call and this note does not argue with it. It does mean the one out-of-band alarm that would have noticed a missing capture is gone.

  2. The Mac's dispatch beacon may not be configured — and no routine can check. HEALTHCHECK_URL_DISPATCH is read from config/.env and absent means silently disabled (scripts/dispatch_workflows.sh:44-53), and RUNBOOK.md's 14-row beacon inventory has no row for it — directly below that file's own warning that "an unconfigured beacon reads as coverage while providing none".

    CORRECTED 2026-10-06 (daily audit, on panel run 35's finding, #220). The sentence that stood here — "the dispatch beacon is not configured … established by DSP-1 (panel run 25)" — was wrong twice over, and the error was mine. DSP-1 established the opposite: that this premise is NOT-VERIFIABLE from a routine container. config/.env is gitignored and has never been committed (.gitignore:3), so it does not exist in a cloud session; I confirmed that directly before writing this correction. No routine can establish whether that variable is set, and this note should never have said one had.

    Why the correction matters more than the fact: this file is read by agents who act on it, so a false "established" is executable. The concrete misfire the panel named is that a later sweep reads it, treats DSP-1's reopen condition (b) as fired, and reopens a row in a capped queue over a fact nobody in that container can check.

    The note's conclusion is unaffected. Whether or not the beacon is configured, the capture was missed and nothing paged — that is empirical, and it held again on 2026-10-06. What is not established is the reason the dispatch beacon was silent.

  3. The digest cannot see captures at all. digest_exceptions (src/thales/execution/fleet_digest.py:1157) has exactly four legs — fleet red, a sleeve with no run record, a dark routine, a NEW failed order. There is no capture leg and no timeliness leg, and the gather never collects capture state in the first place. Today is a Monday, and the email policy is weekly-Friday-plus-exceptions, so even a digest that runs correctly mails nothing about this.

  4. Capture QA cannot see the capture instant. capture_qa tests per-date row counts and never reads captured_at — CQA-3, open since panel run 25, on the evidence of 2026-09-14 and 2026-09-15, when every row was taken at 18:24 ET and 18:03 ET and both days scored perfect.

3. The harm, stated honestly

Bounded, and not a trading risk. No sleeve depends on this stream; the live path imports only prices and VIX.

The cost is to the forward-capture moat. If the backup cron lands tonight at the hour it landed on 10-02 (22:53Z = 18:53 ET), today becomes the third post-close skew day after 09-14 and 09-15 — stale, wide quotes wearing an in-session label, and passing every gate, because of layer 4. If it does not land, today is a permanent hole in a stream that has no heal path (contrast the equity ledgers, which healed from broker truth over the September outage). Either way it is unmarked in CAPTURES.md.

Timing matters here: BTB-1's 60-trading-day clock stands at 50 elapsed (60th = 2026-10-19) against 39 panel days accrued, 11 missing. Days are the scarce input, and this is the class of loss that takes them without saying so.

The pattern is not new and that is the point. DSP-1's own table recorded a missing skew dispatch on 2026-09-14 and 2026-09-15 (single post-close commits at 22:29Z and 22:10Z) and a missing vrp dispatch on 09-16. DSP-1 was dismissed at panel run 25 on the ground of "zero realized harm, and the backstop worked every single day" — which was true of trading, where the backstop is a late trade inside the session. For the capture, the backstop is a post-close observation, which is a different thing from a late one. Today is at least the third occurrence, and it is the first to happen with thales-skew already deleted.

4. Preserved from PR #210, at the owner's request

#210 was closed (not merged) on 2026-10-03 06:27:30Z. The owner's close comment:

"The diagnosis here was correct (an early ping does not satisfy a cron check's later slot). If healthchecks is re-introduced, use this PR's 30 14 reasoning when re-creating the checks."

The record that carried that reasoning rode the closed branch and never landed, so it is restated here in full:

Mechanism. healthchecks sets a cron check's next deadline to the first cron occurrence after the last ping. A ping landing earlier in the day than the check's own expected minute therefore leaves that day's slot unsatisfied, and the check alarms one grace period later. The three sleeve checks were created with the staggered GitHub crons 35 / 40 / 45 14 * * 1-5, grace 8h; the executors then moved twice (2026-09-02, the Mac's dispatch to 14:35Z for all three; 2026-10-02 #201, GitHub's legs to 15:30–15:40Z) and the checks moved neither time. Arithmetic, matched to the second on 2026-10-01: ping 14:37:16Z → next 45 14 occurrence 14:45Z → +8h grace → DOWN | thales-vrp at 22:45:00Z.

The forecast, and its confirmation. The 10-02 audit predicted a DOWN | thales-meanrev at ~22:40Z and a DOWN | thales-vrp at ~22:45Z that night as a falsifiable test of this mechanism. Both arrived: 22:40:03Z and 22:45:00Z. The mechanism is confirmed, not merely plausible.

If the checks are re-created, give each a cron of 30 14 * * 1-5, TZ UTC, grace 8h: ≥5 minutes of margin before the 14:35Z dispatch slot, while an 8h grace still alarms the same evening (22:30Z) and still absorbs a GitHub backup leg delayed to ~20:00Z. Note that momentum's old 35 14 was never safe either — it had zero margin and survived only because its run takes ~15 minutes.

5. What was not done, and why

No detector was built. The fix that would make this loud — a capture-absence or capture-instant leg in the digest's exception list — needs capture state wired into the digest gather, which collects none today, and it lands on top of CQA-3, whose own header marks it "not approvable for build until you rule on the 2026-09-02 B4 append". Building it tonight would both exceed one careful session and step over the owner-only approval gate at research/queue/approved/. It is escalated instead, with the ask stated as a ruling plus an approved row.

Carried in the 2026-10-05 daily audit report.


Append 2026-10-06 — day two, and the regression has a boundary

The daily audit of 2026-10-06 found the same silence again, and narrowed it.

Day two of the same loss. No skew-snapshot.yml run of any kind existed for 2026-10-06 as of 22:08Z — neither the 19:00Z Mac dispatch nor the 19:00Z backup cron, the latter running against a day when GitHub's scheduled events were ~4.9 h late across the board. 2026-10-05's capture did eventually land, via the backup leg, at captured_at 2026-10-06T00:43:01Z = 20:43 ET, 4h43m after the close — so 10-05 is a post-close observation wearing an in-session label, and 10-06 is, so far, absent.

The regression has a start date, which the one-day view could not see. Read off captured_at in data/options_skew.jsonl, the in-session dispatch worked on eleven consecutive sessions, 09-16 through 10-02 (every capture instant 19:01–19:16Z = 15:01–15:16 ET, an hour before the close, exactly as intended), and then stopped. This is not a chronically broken agent; it is a working agent that regressed between Friday 10-02 12:01 PT and Monday 10-05 12:00 PT.

Two candidate causes are ruled out from here.

  • Not gh authentication, and not a dead launchd subsystem. The trading dispatch fired at 14:35:06/09/11Z on 10-05 and 14:35:06/09/13Z on 10-06 — inside six seconds, both days. dispatch_workflows.sh fails closed on gh auth status for all workflows (lines 85-90), so a broken token would have taken trading with it.
  • Not the deleted healthchecks check. thales-skew was deleted 10-02 (#212), which is suggestively close to the regression boundary, but the script's beacon() swallows a failed ping into a log WARN (lines 49-54) and cannot abort a dispatch. A deleted check cannot break the dispatch that reports to it.

That leaves the installed state of com.thales.dispatch-skew on the Mac, or the machine being asleep at midday — neither visible from a cloud routine, and nothing in the repo touched ops/launchd/ or the dispatch script in the 30 days to 2026-10-06.

No detector was built, and this time the reason is not the approval gate. thales captures carries stale_after_td=2 on the skew streams by deliberate design (src/thales/data/captures.py:59-84), so a single missing session reads stale 1 / status OK. That is a tolerance, not a defect — it exists so a late-but-same-night capture does not cry wolf, and repointing it would trade one false-alarm class for another. The missing capability is a different thing: a time-gated stream with no row for a session that has already closed, which is new behaviour in the digest's exception list, and that remains behind the owner-only gate at research/queue/approved/. Recording the distinction so the next session does not mistake the tolerance for the bug.

The likely remedy is already scheduled. The tbx-win cutover's agent fires weekdays 18:00 PT from 2026-10-06 (#214), and on a pass the Windows box takes over both dispatches, skew at 12:00 local. Its gate 1 reads the box's own practice logs (scripts/win/practice_gate.py), not the Mac's dispatches, so the Mac's regression neither blocks nor is hidden by it. The consequence worth stating: if the cutover passes, this finding self-retires; if it postpones, the capture stream stays dark and nothing reports that.


Append 2026-10-07 (panel run 36) — the regression boundary is not real, and 10-06 was not a lost day

Two corrections to the 10-06 append above, both measured, and they point in opposite directions: the dispatch problem is older and wider than it says, and the capture loss is smaller than it says. Neither is a quibble — the first sends a diagnosis at the wrong machine, and the second is the lead reason the 10-06 daily audit gave for calling an INCIDENT.

1. "Eleven consecutive sessions, 09-16 through 10-02 … then regressed over one weekend" — the count is right, "consecutive" is false, and the conclusion inverts

The authoritative witness is the Actions API partitioned by event, which this file already queries for its 14:35Z timestamps. Over the 23 NYSE sessions 2026-09-03 → 2026-10-06:

dispatchfiredmissed
trading, 07:35 PT (paper-trading.yml)22 of 2309-14
skew, 12:00 PT (skew-snapshot.yml)17 of 2309-14, 09-15, 09-21, 09-23, 10-05, 10-06

Eleven workflow_dispatch runs do fall in 09-16 → 10-02 — but that window holds thirteen sessions, and 09-21 and 09-23 have no dispatch leg at all. The eleven are 11 of 13, not eleven in a row. So the window cited as proof of health contains two instances of the very failure it is cited against, and there is no 10-02 → 10-05 boundary: the identical signature (trading dispatch fires, skew dispatch does not, a single post-close schedule leg) occurred on 09-15, 09-21 and 09-23 — all before the thales-skew healthcheck deletion of 10-02 that the append calls "suspiciously close to the regression boundary".

What the numbers say instead: a 26% miss rate on the 12:00 PT slot, sustained for a month, against 4% on the 07:35 PT slot from the same agent-pair on the same machine. On 09-15, 09-21 and 09-23 the trading dispatch fired and the skew dispatch did not — same machine, same day, about four and a half hours apart.

Why the correction changes the next action. The append's own next paragraph sends the reader to "the installed state of com.thales.dispatch-skew … or the machine being asleep at midday", and the 10-06 daily audit's escalation pre-wrote the remedy as one command, launchctl list | grep thales. A misinstalled or unloaded agent misses 100% of its slots, not 26% — so that command will very likely show the agent present and loaded, and the reader will conclude nothing is wrong. Likewise, hunting for a discrete change over the 10-02/10-05 weekend will find nothing, because this file already states (:191) that nothing in the repo touched ops/launchd/ or the dispatch script in 30 days. The signature that is measured — midday-only, intermittent, a month long, with the 07:35 slot near-perfect — points at the machine not being reliably awake at noon, which RUNBOOK.md:34-35 already documents as the accepted degraded mode ("Mac asleep → nothing dispatched; the late cron runs as before"). The discriminator is the Actions API partitioned by event, not launchctl.

This file already had it right, seventy lines up. :101-108, merged as #219, reads: "The pattern is not new and that is the point. DSP-1's own table recorded a missing skew dispatch on 2026-09-14 and 2026-09-15 … Today is at least the third occurrence." The 10-06 append neither cites nor supersedes it, so one file now carries both readings with the later text holding the wrong one. The sibling append merged seven hours earlier the same morning (#220's cqa-3) independently lists 09-14 and 09-15 as all-post-close days of exactly this shape.

Second-order cost, which is the part that outlives the cutover. A 26% structural loss of the in-session capture instant is a standing data-character fact bearing on the B4 threshold countersign #220 just re-measured. "A two-day regression about to self-retire" reads as a tail event and invites deferring that ruling; "26% for a month, cause not yet fixed" does not.

2. 2026-10-06 was NOT a permanent hole. It landed 41 minutes after the audit said it had not — and it rescued its week

:167 — "10-06 is, so far, absent" — was true at 22:08Z and is false at HEAD. It is corrected here rather than edited because the 10-06 daily audit leads its INCIDENT verdict with "a second permanent notch — a capture day lost is unrecoverable … today is a new loss", and that is the claim a later sweep acts on.

skew-snapshot.yml ran as a schedule event from 22:57:05Z (3 h 57 m past its 19:00Z cron) and committed as 50005e7 at 23:03:56Z. The capture instant is 2026-10-06T22:57:52Z = 18:57 ET, +178 min past that session's NYSE close. The audit email left at 22:16:34Z — 41 minutes early.

It was four streams, not none. 50005e7 added options_chains/2026-10-06.parquet (1,334,345 B), options_iv_term/2026-10-06.parquet (106,012 B), 882 rows to options_skew.jsonl, and shortability/2026-10-06.parquet (247,556 B). So check 7's "no skew / FINRA / shortability capture for 2026-10-06" is wrong on skew, on shortability, and on the chains/term twins.

The fourth named stream was not due. FINRA short-volume lands on a one-session lag: skew: snapshot 2026-10-01 added short_volume/2026-09-30, snapshot 2026-10-02 added 2026-10-01. The 10-05 run added both 10-02 and 10-05 only because it itself fired at 00:49Z (20:49 ET), late enough to catch FINRA's publication; the 10-06 run fired at 22:57Z (18:57 ET), before it. So short_volume/2026-10-06 is expected in the 10-07 run. (pit_snapshots reading 10-05 is likewise correct and already owned — that stream is written by the next session's pre-open run, per STAMP-3.)

Zero of three sessions lost since the boundary this append proposed:

sessioncapture instantvs NYSE closerows
2026-10-0219:01:11Z (15:01 ET)in-session, −59 min282
2026-10-052026-10-06T00:43:01Z (20:43 ET)post-close +283 min282
2026-10-0622:57:52Z (18:57 ET)post-close +178 min882

And 10-06 is the stronger observation, not the weaker one. B4 admits the last date per ISO week carrying ≥400 valid rows. From the store:

  • ISO week 40 (09-28 → 10-04): 10-01 = 281, 10-02 = 282 → no qualifying date; a lost week, already owned by #198.
  • ISO week 41 (10-05 → 10-11): 10-05 = 282, 10-06 = 882 — the week's only qualifying date so far.

The late backup leg did not merely avoid a hole; it is what keeps W41 from becoming another lost week. Calling it a loss inverts its sign.

Why the rows jumped 282 → 882, since the coincidence invites the wrong inference. Not the lateness — the expiry calendar, exactly as #198 pre-computed from a 20–45 DTE window straddling the monthly expiry gap. The November monthly (2026-11-20) enters that window to the day:

sessionOct-monthly DTENov-monthly DTENov in [20,45]?rows
2026-10-011550no281
2026-10-021449no282
2026-10-051146no, by one day282
2026-10-061045yes882

10-05 misses by a single day. Neither row count should be read as evidence about the dispatch regression in either direction.

What is NOT corrected, so this append is not read as an all-clear

The in-session dispatch is still broken, and 10-06 is the third consecutive session captured post-close. No instrument the shop owns sees it: thales capture-qa returns exit 0 with "all capture streams pass integrity QA", thales captures reads the skew family current, and its interior-gap warning lists only 09-21…09-30 — because a file exists for every other date. The cost is to data character, which open/cqa-3 owns and where the base rate is now 16 of 38 days entirely post-close (42%), up from the 15 of 37 #220 pinned. The cost is not, on this evidence, to coverage.

Three citation defects in the same append, corrected here at zero cost

  1. "dispatch_workflows.sh fails closed on gh auth status for all workflows (lines 85-90)" — the gate is at 65-71 (if ! "$GH" auth status at 67, exit 1 at 70). The file is 88 lines, so 89-90 do not exist; 83-88 are the beacon tail, which pings and returns regardless. The substance is right (one global check before the loop, nothing dispatched) but a session re-verifying "does it fail closed?" lands on the beacon call and can conclude the opposite.
  2. "the trading dispatch fired at 14:35:06/09/11Z on 10-05 and 14:35:06/09/13Z on 10-06" — the two days are swapped, and the 10-06 triple is a hybrid. The API gives 10-05 = 14:35:08 / :10 / :13 and 10-06 = 14:35:06 / :09 / :11. The derived claim survives (both days span 5 s, all three workflows dispatched); the digits do not.
  3. "stale_after_td = 2 … so a single missing session reads stale 1 / status OK" — the citation (captures.py:59/71/84) is exact, but the gate is elif stale_td > s.stale_after_td (:499), so stale_td of 1 and 2 both read OK and three consecutive missing sessions are needed to go STALE. The tolerance is twice as wide as the paragraph whose job is to state it.

One NOT-VERIFIABLE retired by direct observation

:203-209 says a postponement leaves the stream "dark and silent", and the cutover's email path depends on EMAIL_ADDRESS / EMAIL_APP_PASSWORD from the absent config/.env — a premise no routine can check. It can now be retired empirically: the cutover's postponement email exists. Sent 2026-10-07T01:44:19Z, subject ⏸ thales tbx-win cutover postponed: a gate did not hold (retrying tomorrow 18:00), carrying gate 1's value verbatim. So a cutover postponement is reported on the owner's own channel every weekday evening, exactly as ops/windows-host/README.md:252 claims; what has no alarm is the capture darkness, which is a different thing and is open/cqa-3's and open/dgx-1's. Recorded so nobody builds a postponement detector that #214 already shipped.

And the branch resolved to the one that needs the owner. That email's gate 1 reads Connection timed out during banner exchange | Connection to UNKNOWN port 65535 timed out — an SSH transport failure, not a practice-day verdict. Gate 2 passed (#206 MERGEABLE). The box is not answering, so "if the cutover passes, this finding self-retires" has resolved to the other branch, and the mechanism is written up in research/queue/dismissed/cut-1-cutover-gate-conflates-unreachable-with-not-ready.md.