Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

BGS-1 — moving the digest to the Windows box gated six jobs behind a new one-minute job, and four detectors that lived inside those jobs have no box-side equivalent; #251's new tests assert what was added and are blind, by construction, to every one of the four removals

status: open · raised panel run 40 (2026-10-19; change-reviewer catch on #251, merged 2026-10-18 — a defect row, not a research or capture proposal, so it does NOT discharge the frontier scout's cadence clock; same treatment as LAB-1, BTB-1 and CQA-3) · class: loudness/safety (P0, G2) with a moat leg (G4) and an instrument-calibration leg (P2/G5 — tests that cannot fail) · effort ~1.25 pd for all four legs, each independently shippable · horizon: legs (a) (b) (c) activate the day an admin installs the box's digest task — latent today, imminent; leg (d) is live now


Plain-language summary for an owner reading one paragraph

#251 is good work and its motivation was right: the Windows box does the real work, so GitHub should stop re-doing it. It does that by putting a new one-minute job, box, in front of each GitHub leg — the job asks main whether the box already committed today's record, and the expensive leg skips if it did. The trouble is what else lived inside those legs. Six jobs are now gated, and four of them carried something the box does not replicate: the only place 552 of this repo's 1,795 tests ever run; the only machine-readable channel that says a sleeve did not trade; the fix for the 2026-10-07 race that emailed "meanrev did not run" while meanrev was mid-flight and went on to submit 62 orders; and — for the gate itself — any alarm at all. None of this is hypothetical bad luck: #251 shipped 18 new tests, and all four removals pass them, because the tests check that the gating was added correctly rather than that nothing was lost. Three of the four controls below were executed; the fourth is a GitHub-semantics claim I could not run here and have labelled a hypothesis rather than asserted.


Leg (a) — 552 of 1,795 tests (31%) and the entire web suite now run NOWHERE on a normal weekday, and the alert for that is exempted

The defect. fleet-digest.yml:243-245 added to the observability job:

  observability:
    needs: box
    if: ${{ needs.box.outputs.ran != 'true' }}

That job is the sole executor of the reporting/research test tier. Enumerated over every pytest invocation in the repo:

$ grep -rn 'pytest' .github/workflows/*.yml scripts/win/*.py
fleet-digest.yml:266      python -m pytest tests/ -q -m "observability or research"   <- the ONLY one
paper-trading.yml:100     -m "not observability and not research"
paper-trading-vrp.yml:117 -m "not observability and not research"
paper-trading-meanrev:132 -m "not observability and not research"
scripts/win/session.py:423 -m "not observability and not research"

(fleet-digest.yml:301 and :304 look like a second executor but sit inside alert-observability's hint: text — a reproduce-locally instruction, not a run step.) The box runs the complement, and only for trading: session.py:421-424 is not observability and not research, and run_task (session.py:555) calls test_gate() only if task == "trading" — so the box's digest task runs no tests at all.

Tier size, measured at HEAD 6c8d2d0:

$ python -m pytest tests/ -q -m "observability or research" --collect-only
552/1795 tests collected (1243 deselected)

The web npm suite (tests/mcp.test.ts, surface completeness, the scale-free contract) lives only in that job and has no other home. alert-observability (fleet-digest.yml:299) is always() && result != 'success' && result != 'skipped', so the removal is silent. And the job's own header is now false on a box-ok day — it reads (fleet-digest.yml:240-241):

"As its own job it still reds the run and still pages via alert-observability below — it just cannot hold the mail hostage."

The same header records why the job exists: on 2026-07-27 a missing matplotlib failed a chart assertion and "killed the digest email for three trading days, taking the fleet's only same-day did-not-run detector with it."

Negative controls (executed by the change-reviewer lens; the executor map, tier size and header text re-verified by the chair). A failing observability-marked test was planted in a copy of the tree and both selectors run:

=== the box's EXACT pre-trade selector (session.py:423-424):
19 deselected in 0.06s                      <- green, nothing ran
=== the fleet-digest observability job's selector:
FAILED tests/test_planted_rendering_bug.py::test_a_rendering_regression_...
1 failed, 18 passed

Second control: both the observability and site-export gates were deleted from fleet-digest.yml and #251's whole new suite stayed green (18 passed). Nothing asserts those gates in either direction.

Harm path. From the day the box's digest task is installed, rendering, digest, lab and web regressions land with CI green and nothing paged. That is the exact class that produced the 10-15 mid-sentence truncation and the 10-16 headline cut (#244, still open) — defects in the one email the owner reads, caught only by this tier. The decision corrupted: the steward and the daily audit both read "all green" as "the reporting layer is tested".

Fix (~0.25 pd) and why it is nearly free. Drop needs: box from observability. Its own header says "No secrets and no data cache: every observability test is fixture-based" and it is capped at timeout-minutes: 10 — so it is the cheap job and was never what drained the Actions allowance #251 was written to protect. Plus one line in the gated/ parity test pinning that this job is not box-gated.


Leg (b) — on a box-ok day, "SLEEVE DID NOT RUN" has no channel any reader outside the Windows PC can see

The defect, verified link by link at HEAD:

  1. session.py:285-287 — the box turns the digest's NO-RUN exit code into success: if rc == 2: log("...NO RUN"); return 0. Its docstring is explicit that this is deliberate: "Exit 2 = the digest is healthy but a live sleeve has no run record today: that is logged, not a failure of the digest." That is correct about the digest and is where the fleet-level consequence begins.
  2. ctx.failed therefore stays empty, so session.py:252 stamps Box-Result: ok.
  3. _box_ran.yml:55 skips only on ok → GitHub's digest job is skipped.
  4. no-run (fleet-digest.yml:321-322, needs: digest, if: needs.digest.outputs.code == '2') is skipped with its dependency.
  5. The CLI's other two channels are dead on the box: src/thales/utils/notifications.py:61-62 returns early unless GITHUB_ACTIONS == "true", so the box cannot write an annotation; and the escalation email is withheld by notifications.email: weekly_digest_only.

Surviving record: one log() line in C:\thales\home\logs\digest.log — not committed, not fetchable, and unreadable by the steward, which under the 2026-10-07 delegation is the owner's stand-in for exactly these withheld records.

Negative control (executed end to end by the change-reviewer lens, with fleet digest stubbed at rc=2 and everything else real — real session.py, real jq filter, real config/fleet.yaml):

[session] fleet digest: a live sleeve produced no run record today (NO RUN)
box Result: Result(sleeve='digest', failed=[], committed=True, detail='')
commit the box left:
   site: daily public data refresh 2026-10-19
   Runner: tbx-win
   Box-Result: ok
_box_ran.yml jq over that commit -> n=1 => ran=true => `digest` SKIPPED => `no-run` SKIPPED
email_allowed(alert) = False
github_annotation outside Actions prints -> (nothing)

grep -rn 'no-run' tests/ returns only CLI exit-contract tests — nothing inspects the no-run job, which is why its consumer disappearing was not noticed.

Harm path. This is the precise failure the no-run job was built to end, in its own words (fleet-digest.yml:306-308): "the digest detected the outage, said so in the email, and still exited 0, so gh run list showed an all-green fleet." Two triggers have already occurred this month: a box slot not firing (the 10-16 pattern) and a sleeve's step failing closed (10-13 meanrev). Mon–Thu the owner gets nothing by design, the steward sees green Actions and no annotation, and if vrp is the silent sleeve an open SPY put spread goes unmanaged with no alarm.

Fix (~0.5 pd). On rc == 2, append to ctx.failed (e.g. no-run) so the commit stamps Box-Result: failed (no-run) → the gate does not skip → GitHub's digest runs as the backup → the existing no-run job reds the run exactly as before. Optionally also call session.alert() on rc==2 (that path has its own SMTP and does not consult email_allowed). Add a test that an rc==2 digest leaves a commit the gate does not skip on.


Leg (c) — the 2026-10-07 false-NO-RUN race fix was never ported to the box, and its test still passes because it reads the YAML of a job that no longer runs

The defect. fleet-digest.yml:124-183 is the "Wait for in-flight sleeve runs, then refresh state" step — a bounded gh run list wait, then git checkout -- data/ and git pull --ff-only. It was built after the 2026-10-07 incident in which the digest emailed "meanrev NO RUN" while meanrev was mid-flight and went on to submit 62 orders. tests/test_execution/test_ops_alerting.py:378-416 pins four of its properties as "each load-bearing". The box replicates none:

box STEPS['digest']   = ['fleet digest', 'site export', 'persist']
YAML digest job steps = [checkout, setup-python, Restore data cache, Install deps,
                         Create config/.env,
                         'Wait for in-flight sleeve runs, then refresh state',
                         'Send fleet digest', beacon]

boot.py:226 syncs main before the task, i.e. at 08:30 PT; there is no post-trade refresh, and the box has no gh at all.

Negative control (executed by the change-reviewer lens). #251's PR body advertises tests/test_digest_on_box.py:68-78 as "parity with fleet-digest.yml's commands". Its regex extracts only thales tokens:

parity test compares ONLY these tokens: ['fleet digest', 'export-public']

The in-flight/refresh step contains no thales invocation, so it is outside the parity contract by construction, and the suite is green with the box missing it entirely. test_no_run_detector_waits_for_in_flight_sleeve_runs also stays green, because it only parses fleet-digest.yml.

Harm path — conditional, and the condition is a day that already happened. Box-traded days are protected by construction (the digest waits on the runner lock behind the 07:35 PT sequence). The exposed day is the one #251 itself cites: a sleeve fails on the box (Box-Result: failed) → GitHub's backup trades it at 15:30Z + queue delay, i.e. after the box's 08:30 PT = 15:30Z digest. The box then renders NO RUN for a sleeve that traded an hour later, swallows it (leg b), commits ok, publishes web/public/data without that trade, and the GitHub digest that would have waited and re-read state is skipped. On a Friday that wrong "NO RUN / 0 trades" is what the owner reads; on thales.report it is what the public page shows. Precedent: 10-13.

Fix (~0.5–1 pd), cheapest first. Move the box's digest slot after the backup legs' observed completion; or port the git pull --ff-only refresh half into STEPS["digest"] as a pre-step (the wait needs gh, the refresh does not, and the refresh is the half that matters); or widen the parity test from "thales tokens" to "every run step accounted for or explicitly waived" — which closes the whole class rather than this instance.


Leg (d) — the gate is unalerted, and at job level it is fail-CLOSED against its own header

The defect. _box_ran.yml:16-18 promises "FAILS OPEN: if the API call fails, ran is false and the backup runs as it always did." True at step level — eight synthetic scenarios (gh exit 1, empty stdout, null, true, garbage, count 0/1, manual dispatch) all resolve to ran=false with the ::warning::could not read main's commits line. But the gated jobs are needs: box + if: ${{ needs.box.outputs.ran != 'true' }} with no always() (paper-trading.yml:28-29, -meanrev:49-50, -vrp:47-48, skew-snapshot.yml:31-32, fleet-digest.yml:41-42), so per documented Actions semantics a job whose needs dependency fails is skipped. The repo knows the rule: site-export and every alert-* job in the same file use always() && …, and site-export's is the only gate written always() && needs.box.outputs.ran != 'true'.

Then skipped is laundered: it had two causes with opposite meanings for the backup's health, and the alert layer is told to ignore both.

causemeansalerted?
gate ran, ran=truethe box did today's work — benigncorrectly silent
gate died (allowance refusal, runner loss, timeout-minutes: 3)GitHub was never asked; the backup did not runsilent — wrongly

skipped was unreachable on these jobs until Sunday. The != 'skipped' clause dates to 2026-08-20 (git log -S on paper-trading.yml returns exactly #105, the runner-loss backstop), when the trade job had no needs: and no if::

$ git show 6c8d2d0^:.github/workflows/paper-trading.yml | sed -n '/^jobs:/,/runs-on/p'
jobs:
  trade:
    runs-on: ubuntu-24.04

Eleven alert jobs carry != 'skipped'; six guard a now-gated job. _box_ran.yml has no alert job anywhere — one unalerted SPOF in front of five workflows.

The capture arm is the uncovered half, and its loss is permanent. skew-snapshot.yml's gated snapshot job contains, in order: Snapshot option skew (full universe) · FINRA short-sale capture · Shortability snapshot · Persist · Capture integrity QA · beacon. A gate death skips all six, so three un-backfillable streams are not captured and the QA that would notice is inside the thing that was skipped. There is no equity curve to fail to advance, so heartbeat-monitor.yml (correctly left ungated by #251 — the sleeves' surviving dead-man) does not cover it; and the four trading/capture healthchecks (thales-momentum, -meanrev, -vrp, -skew) were deleted on healthchecks.io 2026-10-02 by owner decision (ops/windows-host/README.md), so there is no out-of-band beacon either. NORTHSTAR.md §1 asset 2: "A missed day is a permanent notch."

Negative control (executed). The suite does not merely leave this unpinned — it pins it open. tests/test_execution/test_ops_alerting.py:94, inside test_alert_job_condition_actually_fires_on_a_lost_runner, parametrised over all six alerting workflows:

assert f"needs.{work_job}.result != 'skipped'" in flat

It is the only assertion in that test with no rationale message — every other carries a paragraph naming the incident it encodes. Line 354 pins the same clause for site-export. python -m pytest tests/test_execution/test_ops_alerting.py -q → 60 passed. Separately, adding always() && to paper-trading.yml's gate leaves #251's 18 new tests green, while inverting the gate is caught — so the test bites on inversion and is blind to the fail-open property the header advertises.

#251's own new test comment documents only the benign cause, which is the blind spot in one line (test_ops_alerting.py:321):

# `box` (2026-10-19): the whole job is skipped when tbx-win already exported today.

What was verified empirically, and the one link that was not. Refusal produces §4c's platform-refusal signature, including on reusable-workflow caller jobs (which box is) — run 39191451973, 2026-10-19 02:45Z:

watch         | failure | runner_id=0 | steps=0
alert / email | failure | runner_id=0 | steps=0

annotation verbatim: "The job was not started because recent account payments have failed or your spending limit needs to be increased." NOT verified here: that GitHub reports a job whose needs failed as skipped rather than failure. No Actions runner is available and act is not installed, and no gated workflow has fired since #251 merged (first scheduled firings 2026-10-19 15:30Z, after this run). Labelled a hypothesis; it is this leg's primary kill criterion below.

Fix (~0.25 pd). Prefix the four gates with always() && (matching site-export, which already does), add one alert-box job per gated workflow (needs: box, if: ${{ always() && needs.box.result != 'success' }}) so the record says "the gate could not run" rather than "the sleeve failed", give line 94's bare assertion the rationale it never had, and assert always() in the gated test.


Pre-registered kill criteria — per leg, so a partial fix closes only its own leg

  • (a) dies when grep -n 'needs: box' -A2 .github/workflows/fleet-digest.yml shows the observability job ungated and a test asserts it stays so; or on an owner ruling that the reporting tier may run weekly rather than daily, in which case the weekly cron is the fix and this leg closes with it.
  • (b) dies when an rc==2 digest on the box produces a record readable off the box — a Box-Result: failed (no-run) commit, a committed ledger row, or an email — demonstrated by a test. It does not die on "the Friday email will say so": leg (b) is about Mon–Thu, and about vrp.
  • (c) dies when either the box's digest runs after the backup legs, or the refresh half is ported, or the parity test accounts for every run step. Also dies if the owner retires GitHub's trading backup legs outright — then there is no later-trading leg to race.
  • (d) dies immediately if a gated workflow firing on or after 2026-10-19 15:30Z with box dead reports the gated job as failure (or anything but skipped): then the alert fires as designed and there is no hole. Checkable from the Actions API by the next panel run. It also dies if CQA-3 is built such that thales capture-qa reds a session with no capture at all regardless of which leg was meant to take it — that closes the moat arm, and what remains is a sleeve-level duplicate of the heartbeat.
  • The whole row dies if the Actions backup legs are retired rather than gated. That is a live option once the allowance question is settled, and it would be a cleaner answer than four fixes.

Adjacent, flagged not claimed

  • DSP-1 (dismissed) — its panel-39 append already says "the Runner: trailer needs a reader — a session that trades from the backup leg while the box is silent is the signal, and it is currently written down and read by nobody." #251 partly satisfies that and deserves the credit: whether the gated job ran or skipped is now a machine-readable record of which runner traded. But the reader it added acts (skips) rather than reports, so DSP-1's channel-neutral reopen condition ("a session trades from the backup leg on any day where no human or agent reader is notified by any channel within one trading day") is made more likely by leg (b), not less. Whether leg (b) belongs here or as a dated append on DSP-1 is the steward's git mv per ops/QUEUE_TRIAGE.md:100-102, not the panel's.
  • LAB-1 (open) — the sibling defect in the same gate family (marker deselection happening after collection). Cited as precedent for treatment, not duplicated.
  • CQA-3 (open) — blind to the capture instant; leg (d) is blind to the capture not happening at all. Kill criterion (d) names the seam.
  • Docs, no row: ops/windows-host/README.md:57 (added by #251) says the digest is a tbx-win task, while :59 two lines later still says "Fleet digest … stay on GitHub — the observers must keep working when tbx-win is down" — same table, opposite answers, and :59's rationale is precisely what legs (a)–(c) are about. CLAUDE.md:388 still describes fleet-digest.yml as running the "public-site refresh, no-run detector, observability suite", none of which it does on a box-ok day. Both are green against test_docs_consistency. ~0.1 pd, and the README row is what an interactive session or the steward reads when the box misbehaves.
  • Minor, no row: session.py:234's new docstring cites scripts/box_ran_today.py, which does not exist; the real gate is _box_ran.yml, cited correctly 18 lines later. And no detector pins the runner image (grep -rn ubuntu tests/ → nothing), so the next workflow a routine adds will default to ubuntu-latest and silently land on Ubuntu 26 — one line in test_ops_alerting.py closes it.

Reviewed in the same pass and found CLEAN

The push-fix half of #251 (BOT identity on every git call, pull_rebase() abort) — each half reverted in a copy and all three of its tests bite, as the commit message claims; correctly observability-marked so it can never fail-close the box's own gate. _box_ran.yml's jq filter — real jq over six commit shapes; Box-Result: failed (trade), a Runner: github-actions commit, another sleeve's subject and a Revert "…" all correctly rejected. The UTC date agreement between the box's subjects and the gate's date -u +%F, holidays included. detect_runner_loss.py (keys strictly on cancelled; skipped jobs add no false positive). The _last_beacon_date_git fallback, which is necessary — without it every routine reads unknown and #249's steward-dark alarm could never fire. #249's detector keys on the literal beacon/steward string in ops/ROUTINES.md:44, so it cannot be disarmed by renaming the routine, as its comment claims. And #248 ON main: 15 files / 20 jobs, grep -rn ubuntu-latest .github/ → none, no composite actions, no matrix strategies, every reusable-workflow caller carrying its own pinned runs-on; the commit message's "14 workflows (19 jobs)" is exact at its parent.

CORRECTION, appended 2026-10-19 14:50Z — "#248 in full" was wrong, and this run's own beacon push is the proof. This row first recorded #248 as complete. It is complete on main, which is what both of this run's lenses greped — and that is the wrong ref for one workflow. routine-beacon.yml fires on push: branches: ['beacon/**'], and GitHub resolves a push-triggered workflow from the pushed ref. All seven beacon/* branches carry a full stale repo tree, and their copy still reads ubuntu-latest:

main:         .github/workflows/routine-beacon.yml:48  runs-on: ubuntu-24.04
beacon/panel: .github/workflows/routine-beacon.yml:48  runs-on: ubuntu-latest

Measured on run 39263175000 — panel run 40's own 13:47Z beacon push, twelve hours after #248 merged:

relay | failure | labels=["ubuntu-latest"] | runner_id=0

Found by the weekly audit (#253, finding ❌3, opened 14:27Z today), which reproduced it independently; the chair verified it above rather than taking it on report. Credit to that audit, and the miss is this panel's: two lenses greped main and neither asked which ref executes.

Honest severity, and why it is not a leg of this row. The relay is a one-step curl and will most likely survive Ubuntu 26, so the image is the small half. The durable defect is that no future change to routine-beacon.yml on main can ever reach the push leg — and the push leg is the only one a cloud routine can use (repository_dispatch is denied to routine sessions, incident 2026-08-10). The liveness relay is frozen at whatever seven stale branches happen to hold. That is a different object from #251's gating and belongs to #253, which recommends recreating the branches from main or making them orphan branches holding only last-run.txt, plus a test asserting each beacon branch's copy matches main's — and correctly declines to do it, since rewriting seven liveness branches is an outward-facing act on the only signal that proves a routine ran.

The slot this costs

open/ held 11 of 12 when this was raised; this makes it 12, at the cap. The four legs were deliberately filed as ONE row rather than four: they share a root cause (the move gated jobs whose detectors have no box-side twin) and a single blind spot (#251's tests assert what was added), and they would be built together. The run's other two candidates were likewise kept out of slots — the capture-instant finding went in as an annotation to RAD-1 (same single owner ruling) and SCR-1 was dismissed with live reopen conditions (dismissed/scr-1-fired-scout-criterion-unrecorded.md).


Activation note, appended 2026-10-19 15:40Z — this row's own trigger has FIRED

The header above said legs (a) (b) (c) "activate the day an admin installs the box's digest task — latent today, imminent." That day is today. The deployment landed and the box ran all four of its tasks on schedule, with real git trailers (not a grep over quoted commit text):

2026-10-19T15:30:23Z  trailers=[tbx-win, ok]  site: daily public data refresh 2026-10-19
2026-10-19T14:58:15Z  trailers=[tbx-win, ok]  state: meanrev paper run 2026-10-19
2026-10-19T14:56:50Z  trailers=[tbx-win, ok]  state: paper trading run 2026-10-19
2026-10-19T14:37:09Z  trailers=[tbx-win, ok]  state: vrp paper run 2026-10-19

15:30:23Z is 08:30 PT — the new digest task firing exactly on its registered slot. So install-tasks.ps1 was re-run, boot.py accepts digest, and session.py is stamping Box-Result. Good news first: the Friday 2026-10-23 digest will send, which was the open question this run pushed to the owner.

And that is precisely what arms legs (a)–(c). From today, a Box-Result: ok record exists every weekday, so _box_ran.yml resolves ran=true and GitHub's digest, observability and site-export jobs skip by design. Leg (a) is therefore live in the strongest form available: today the reporting/research tier ran nowhere at all — the box's digest task runs no tests, and GitHub's observability job could not run even if ungated, because the account-level refusal is still in force (every run through 15:30:28Z is runner_id=0, including Main Push Watch on the box's own digest commit).

The two causes are independent and both must be fixed: the refusal lifts by itself (~2026-11-01), after which the gate keeps that tier skipped every day until leg (a) is addressed. Nothing about today's successful deployment closes any leg of this row; it starts their clock.

One honest consequence for leg (d)'s kill criterion: it asks whether a gated job whose box dependency failed reads skipped or failure. Today could not answer it — the box succeeded, so the gate's fail path was never taken on a live firing. The criterion stands unresolved and still cheap, and the next panel run should read it off the first firing where the box record is absent.

ANNOTATION 2026-10-20 (panel run 41) — LEG (d)'s KILL CRITERION IS ANSWERED: skipped, on all five workflows, with the alert skipped too. The criterion was answerable from 2026-10-19's own firings and run 40 read the wrong signal. Plus a new leg (e): the box's own alarm is withheld on the written premise that THIS leg covers it

Annotation only; the body above is untouched.

Leg (d) — CONFIRMED, not hypothesis. Seven gated jobs, five workflows, one evening

Run 40 closed with: "it asks whether a gated job whose box dependency failed reads skipped or failure. Today could not answer it — the box succeeded, so the gate's fail path was never taken on a live firing."

That is wrong, and the way it is wrong is the finding. The box succeeding sets ran; it says nothing about whether the box / check job fails. On 2026-10-19 the gate job failed on every one of the five workflows that call _box_ran.yml — the account-level refusal takes the runner before any step runs — so the fail path was taken five times that evening. Read off the Actions jobs API (conclusion, not inference):

runworkflowbox / checkgated jobits alert
39321205783 (20:40:02Z)Paper Tradingfailuretrade skippedskipped
39321409442 (20:41:27Z)Paper Trading (meanrev)failuretrade skippedskipped
39321400775 (20:41:23Z)Paper Trading (vrp)failuretrade skippedskipped
39337754309 (22:50:52Z)Skew Snapshotfailuresnapshot skippedskipped
39321763264 (20:43:57Z)Fleet Digestfailuredigest, observability, no-run skippedalert, alert-observability skipped

So: skipped. And because every alert: job tests needs.<work>.result != 'skipped' (paper-trading.yml:447, -meanrev.yml:259, -vrp.yml:208, skew-snapshot.yml:151, fleet-digest.yml:290,413), the alarm skips with it. Leg (d) is live and fleet-wide, not digest-local: the three sleeves' backup trading legs and the skew capture are inside it.

The negative control ran itself, in the same run, in the same minute

fleet-digest.yml:337-339 is the control, and it is the only gated job in the repo that carries one:

  site-export:
    needs: [box, digest]
    if: ${{ always() && needs.box.outputs.ran != 'true' }}

In run 39321763264, with the identical failed box / check upstream, site-export → failure (it ran, then hit the refusal) and alert-site-export / email → failure (it fired). The six gated jobs without always() → skipped, silent. Same commit, same gate, same minute: the single token always() && is the whole difference between a loud day and a silent one. No code change was needed to run this control.

The mechanism is documented in this repo, 150 lines below the job that omits it

paper-trading-vrp.yml:200-205, on the alert job:

always() && is MANDATORY: a job if: with no status-check function carries an implicit "all needs succeeded" and would never be reached.

paper-trading-vrp.yml:48-50, the trade job in the same file:

  trade:
    needs: box
    if: ${{ needs.box.outputs.ran != 'true' }}

The knowledge was in the file. The gate was written without it. Identically at paper-trading.yml:27-29, -meanrev.yml:50-52, skew-snapshot.yml:32-34, fleet-digest.yml:40-42 and :244-246.

Fail-CLOSED at job level, against the gate's own capitalised header

_box_ran.yml's header promises: "FAILS OPEN: if the API call fails, ran is false and the backup runs as it always did." The step honours that — || n="", a ::warning::, ran=false. The job does not: when the job dies before writing its output (refusal, runner loss, the 3-minute timeout-minutes, a set -uo pipefail abort), every consumer's implicit "all needs succeeded" skips the leg. The header describes the step's contract and is read as the system's.

Why run 40 misread it — the ambiguity has already claimed a reader

Run 40 flagged that skipped "now has two opposite causes." It does, and the two are indistinguishable in the job list: box-ran-fine → skipped by design; gate-broke → skipped by failure. Run 40 checked the box's commit trailers (which showed the box fine) and concluded the fail path was untaken, while five gate jobs were failing that same evening. The first thing this ambiguity cost was the panel's own reading of its own kill criterion. That is the strongest available argument that the two causes must not share a word.

Leg (e) — NEW. The box's own alarm is withheld on the written premise that this leg covers it, and both halves are dark right now

scripts/win/boot.py:179-184, docstring of send_alert, and the code under it:

"Withheld under the weekly-digest-only policy: the GitHub backup leg covers a box that cannot start."

if email_policy() != "all":
    log(f"alert withheld from email (owner policy): {subject}")
    return

ops/windows-host/README.md:41 names the same compensating control: "If an update ever reboots the box during a run, GitHub's backup leg covers the session."

The compensating control is the leg this row shows is skipped-and-silent whenever box / check fails. Under the refusal that is every firing, every workflow, every day. So for the nine trading days 2026-10-20 → 2026-10-30 (the refusal lifts ~2026-11-01; the September refusal ran 09-21 → 09-30, consistent with a 1st-of-month reset), if the box misses a slot: the box does not mail, because the backup covers it; and the backup does not run and does not alert, because its gate failed. Neither automated alarm speaks (see the correction below for the two after-close readers that do). The daily audit of 2026-10-19 raised the withholding as NEEDS-OWNER and described the backup as "refused" — understating it: a refused leg that reported failure would have alerted. It reports skipped, which the alert layer exempts by design.

Not hypothetical: the steward's 2026-10-16 record has the box missing its second slot that week (skew, past the B4 grace), later traced to a push with no committer identity (win/rebase-identity-fix, folded into #251). A box that misses a slot inside this window is a demonstrated failure mode, not a tail.

CORRECTION TO THIS ANNOTATION'S OWN FIRST DRAFT, and it matters. I drafted this as "neither channel speaks" and as a new NEEDS-OWNER line for Friday. Both were wrong, and the steward's 2026-10-19 record (research/steward/2026-10-19.md:104) is why:

"NEEDS-OWNER: decide whether the box's failure emails stay switched off while GitHub's backup leg is down — the code justifies the silence by saying 'the GitHub backup leg covers it', and until ~11-01 that leg cannot run. — My recommendation: leave notifications.email: weekly_digest_only as it is … A box failure is still found the same evening: the daily audit checks at 3pm PT and I check at 4pm PT, both reading the box's commits. The cost is that a failure is found after the close instead of during the session … The one exposure is vrp's open SPY put spread (expires 11-20), which no one would manage that day."

So: the ask already exists, dated before mine, with a recommendation and a priced residual — and it names a third channel I had missed. The daily audit (~22:00Z) and the steward (~23:00Z) are claude.ai routines that read the box's commits out of git; they depend on neither the gate nor healthchecks, which is why they are still working while everything on GitHub is refused — this panel run is itself the evidence. Detection therefore does not vanish, it moves from during-session to after-close, exactly as the steward priced it.

Two consequences, and I am taking both:

  • No new owner ask from this run. The steward is explicitly rationing (:119: "NEEDS-OWNER today because there are four asks already"). A fifth line restating its second would spend the one channel NOL-1 just finished repairing. The correct move is to leave ask #2 standing and attach this row's evidence to it.
  • This annotation's live-exposure claim is narrowed accordingly. What is dark for the nine trading days 2026-10-20 → 2026-10-30 is the automated, same-session alarm path — the box's email (withheld) and the backup leg's alert job (skipped). What survives is two after-close readers. The blackout-window half of leg (e) is therefore already owned and answered by the steward; what this row adds is below, and it is the half that outlives 2026-11-01.

Leg (e), the part that is NOT already owned — the hole survives the blackout

Everything the steward and the 10-19 daily audit wrote about this is scoped to the refusal window ("until ~11-01", "~9 trading days"). The gate defect is not. After the allowance resets, box / check succeeds and the leg works — until the next time that job fails for any reason, at which point the same silent skip recurs for that day, with no blackout to explain it and nobody looking. Causes that need no billing outage: a transient GitHub API error on the one gh api call, runner loss, the 3-minute timeout-minutes, a set -uo pipefail abort before the output is written. Precedent for exactly this class of one-API-call job crashing transiently is open in this repo right now: PR #245 — the main-push watcher crashed on a transient API error.

That is why the fix below is worth its 0.15 pd even though the loud half of the problem expires by itself in nine trading days.

Leg (e), second part — the documented cover for the DELETED skew healthcheck is two mechanisms that both sit inside gated jobs, leaving one reader

This is the part that compounds, and no single record states it. Trace what watches a missed skew capture — the asset NORTHSTAR §1 calls irreplaceable, where a missed day is a permanent notch:

  1. Its own external alarm is gone. RUNBOOK.md:110: "HEALTHCHECK_URL_SKEW — CHECK DELETED 2026-10-02 … a missed capture day is still unrecoverable at any price, but nothing external watches for one; cover is the fleet digest's no-run detector + thales capture-qa." The owner's call, recorded; the question is only whether the named cover exists.
  2. thales capture-qa is inside the gated job. skew-snapshot.yml:117-122 runs it as a step of snapshot, and snapshot is gated at :32-34 with no always(). Confirmed skipped in run 39337754309.
  3. The no-run detector is inside the gated job too, and fails the other way on a good day: fleet-digest.yml:313-315's no-run needs digest (confirmed skipped), and on a box-ok day the box's own digest runs, where leg (b) shows session.py:285 turns the NO-RUN exit 2 into Box-Result: ok.
  4. healthchecks.io as a layer sends nothing. ops/ROUTINES.md:215-217: "healthchecks.io silent since 2026-10-07 — no notification channel, so a missed ping alarms nowhere; the fleet digest's DARK detection and the steward are the live readers." RUNBOOK.md:82 is blunter: "healthchecks.io sends NOTHING."

So of the two readers ROUTINES.md names, the first cannot speak in either branch — gate fails → skipped; box fine → Box-Result: ok. The steward is the only surviving channel, and it reads run annotations that _alert.yml writes (_alert.yml:88-94) from alert jobs that are themselves skipped. It caught the 10-16 skew miss by reading records, which is the system working — but by inspection, not by alarm, and it was never designed as the sole channel.

This is not a new row and not a new idea; it is the blast radius of leg (d) plus leg (b). It is recorded here because the fix priority follows from it: always() && on skew-snapshot.yml:34 is the cheapest single token in this repo that restores an alarm on a permanent-loss asset.

Kill criterion: falsified if any channel outside the gated jobs can be shown to report a missed capture day — name it and this part dies. The check is a grep for a capture-miss reader outside skew-snapshot.yml's snapshot job and fleet-digest.yml's digest/no-run jobs; I found none.

The instrument-calibration part: the suite asserts this exact rule in two places and in none of the seven that now break it

The sharpest version of leg (d). tests/test_execution/test_ops_alerting.py already encodes the rule, in a comment that could be the fix's own commit message (:76-81):

    # A job-level `if:` with NO status-check function carries an implicit
    # "all needs succeeded", so the comparison would never be evaluated.
    assert "always()" in flat, (
        f"{wf}: `if:` must contain always(). Without it the alert job inherits "
        f"an implicit success requirement and can never run after a failure."
    )

It asserts always() on every alert job (:71-85, parametrized over ALERTING_WORKFLOWS) and on site-export (:324). It asserts it on none of the seven work jobs the new gate feeds. And test_site_export_is_its_own_alerted_job_not_a_swallowed_step was edited by #251 itself — :321 carries the new comment "# box (2026-10-19): the whole job is skipped when tbx-win already exported today." and :322 was widened to assert se.get("needs") in ("digest", ["box", "digest"]).

So the author of #251 opened the one test that asserts this rule, updated it for the gate, preserved always() on the job that had it — and added six sibling gated jobs without it, in the same PR, passing all 18 new tests. This is BGS-1's thesis in one file: the tests assert that the gating was added correctly, and the rule they already encode was never pointed at the jobs the gating was added to.

Which makes the repair cheap and symmetric, and it is the fix I would put to the owner first. ALERTING_WORKFLOWS (:33-40) is already the exact (workflow → gated job) map:

ALERTING_WORKFLOWS = {
    "paper-trading.yml": "trade",
    "paper-trading-meanrev.yml": "trade",
    "paper-trading-vrp.yml": "trade",
    "fleet-digest.yml": "digest",
    "skew-snapshot.yml": "snapshot",
    "heartbeat-monitor.yml": "heartbeat",   # not box-gated
}

One new parametrized lint over that map — skip the entry whose job is not box-gated, assert always() in the rest — plus an explicit case for fleet-digest.yml's observability (the one gated job absent from the map). Then the one-token yml change in seven places. ~0.15 pd total, and the lint is what makes it stay fixed; the yml change alone would regress on the next workflow somebody gates.

Negative control for the lint (pre-registered): strip always() && from fleet-digest.yml:339's site-export — the one job that currently has it — and the new lint must go red. If it stays green it is testing the wrong jobs, which is the defect this annotation is about, reproduced in its own fix.

Both controls EXECUTED this run (the earlier draft only proposed them)

Control 1 — the defect is invisible to the suite that owns the rule. With all seven gated jobs exactly as merged:

$ python3 -m pytest tests/test_execution/test_ops_alerting.py -q
60 passed in 18.75s

Control 2 — the test is not broken; it is aimed at 2 of 9 jobs. Strip always() && from the one gated job that has it (fleet-digest.yml:339, site-export), change nothing else, re-run the same file:

FAILED tests/test_execution/test_ops_alerting.py::
       test_site_export_is_its_own_alerted_job_not_a_swallowed_step
tests/test_execution/test_ops_alerting.py:323: AssertionError
1 failed, 59 passed in 16.13s

Same omission, same file, same pytest invocation: red on the one job the lint covers, green on the six it does not. The working tree was restored immediately (git status --porcelain .github/workflows/ clean); nothing was committed.

That pair is the whole row in two commands. The rule is encoded, the assertion is live and does fail when violated, and it is pointed at two of the nine jobs that need it — so #251 could add six violations and ship green. The repair is to widen the lint's aim, not to write a new one.

One dated consequence for another open row, found by this run's queue lens

NOL-1's pre-registered kill criterion fires on Friday 2026-10-23 and needs a digest to actually send to fire at all. Since #251 the Friday send's primary is the box (ops/windows-host/README.md:57, task \thales\digest, Mon–Fri 08:30 PT), and the box does mail it — scripts/win/session.py:283 runs thales fleet digest --fail-on-no-run --email. GitHub's digest job is the backup, and per leg (d) it is skipped whenever box / check fails, which is every firing under the refusal.

So Friday's digest has exactly one carrier and no working backup. If the box misses Friday, NOL-1's criterion neither fires nor lapses — it stalls, and that row's expiry silently rolls a week. Recorded here rather than in NOL-1 because the cause is this row's gate, not that row's renderer.


ANNOTATION 2026-10-20, 16:00Z (panel run 41, self check-in; its "live instance" WITHDRAWN 2026-10-21 by panel run 42 — see the first subsection) — a THIRD reason the compensating control does not exist: the GitHub backup leg is, in practice, a POST-CLOSE leg

Annotation only; the body above is untouched. Written from a self check-in a few hours after this run's PR was opened.

The "live instance" — WITHDRAWN IN FULL, 2026-10-21 (panel run 42)

This subsection claimed "No sleeve traded on 2026-10-20." That was false. All three sleeves traded on time, on the box: vrp 3ac4abe 14:36:52Z, momentum 4605c2d 14:56:31Z, meanrev 7d6f368 14:57:46Z — every one carrying Runner: tbx-win and Box-Result: ok, pushed together at 14:57:55Z, a full hour before the 15:58Z reading that called them 57–81 minutes overdue. All three equity ledgers end 2026-10-20 (13652.81 / 10787.4 / 9960.87). Each commit landed inside the very window this annotation measured for it, so its own baseline table acquitted the box and was misread.

The cause was a stale clone. Run 41 started on its 13:00Z cron and ran git log at ~15:58Z against a working copy it had never re-fetched — a ~13:00Z view of the repository, predating all five of that day's commits. The daily audit caught it the same night (#259) and the steward confirmed it independently on a freshly fetched main. Verified a third time here before withdrawing, on a fetched clone at 9e28a45.

Nothing in legs (d) or (e) rests on this. Both stand on the 2026-10-19 firings, which are independently verified; the withdrawn reading added no evidence to either. The post-close-latency finding below is also unaffected — it rests on measured created_at values from the Actions API, not on any clone. Removed rather than kept-with-a-caveat because a false event in a row's permanent evidence base is worth less than nothing: it would be cited later by someone who did not read this paragraph.

The new finding: the backup leg lands after the close on most days

I went to check whether today's backup legs ran or skipped — leg (d)'s second confirmation — and could not answer it, which is itself the finding. CLAUDE.md warns never to judge a cron late without reading the actual history first, and it is right. paper-trading.yml's configured cron is 30 15 * * 1-5. Measured created_at for its last eight scheduled runs:

datefireddelay vs 15:30Zvs 20:00Z close
10-1920:40:02Z+5h10mafter close
10-1618:53:45Z+3h24min session
10-1519:18:01Z+3h48min session
10-1419:33:23Z+4h03min session
10-1320:46:03Z+5h16mafter close
10-1222:32:18Z+7h02mafter close
10-0920:12:27Z+4h42mafter close
10-0820:45:11Z+5h15mafter close

Shared-runner queue delay of +3h24m to +7h02m, and 5 of 8 runs were created after the 20:00Z market close. So at 15:58Z today's backup leg is not missing — it is not due yet, and would be expected somewhere between ~18:55Z and ~22:30Z. Leg (d)'s second confirmation is therefore pending, readable from the Actions API this evening by whoever looks next; nothing about it is decided yet, and I am recording that rather than guessing.

But the latency is the point. scripts/win/boot.py:181 withholds the box's alarm because "the GitHub backup leg covers a box that cannot start", and ops/windows-host/README.md:41 says the same of a mid-run reboot. On the measured distribution, that leg does not start until hours after the session it is meant to cover, and on the majority of recent days not until after the close. paper-trading.yml:226 then runs thales run --skip-market-check, so when it does fire late it trades anyway rather than declining.

So the named compensating control fails for three independent reasons, only one of which this row previously had:

  1. the always() gate defect — a failed box / check skips the leg silently (leg (d), confirmed 2026-10-19);
  2. the account-level Actions refusal — nothing runs at all until ~2026-11-01 (leg (e), first half, owned by the steward's ask #2);
  3. queue latency — even fully working and fully funded, the leg is a post-close leg on most days (new, today).

Fixing (1) and waiting out (2) still leaves (3). A same-session backup for a box failure is not what this leg is, and the two places that promise it should stop promising it.

Deliberately NOT proposed

I am not proposing a fix for (3), and not opening a row for it. Moving the backup cron earlier collides with #201's ordering invariant (the crons sit later so they can never race the box) and with the 2026-11-02 DST inversion that LAB-1 leg 3 is about — three constraints on one set of four cron lines is an owner-level design question, not a panel edit. What this annotation asks for is narrower and free: the two sentences claiming the backup "covers" a box failure should say what it actually does, which is to re-run the day late, after the close more often than not.

Kill criterion for this annotation. It dies if the measured delay distribution shifts back inside the session — re-measure created_at minus the configured cron over the next eight scheduled runs; if the median lands before 20:00Z and no run lands after it, (3) is not real and only (1) and (2) stand. It also dies if someone shows thales run --skip-market-check declines to trade post-close by another path, since then the late leg is inert rather than late. Re-measuring costs one API call.

Correction to this run's own 15:02Z owner notification — ALSO WITHDRAWN, 2026-10-21

That notification told the owner that no sleeve had traded. There was no miss, so it was false in its premise and is withdrawn in full — including the paragraph that stood here defending its conclusion as "unchanged and correct." There was nothing for an automated channel to report; detection did not "fall to" the 22:00Z audit; what that read found was a clean 5-of-5 box day and a false alarm in the record. The subsidiary error the paragraph did own stays true but no longer matters: the backup legs had not fired by 15:02Z, and the measured distribution above puts them hours later. The steward's 2026-10-20 record carries "ignore the false 15:02Z alarm" to the owner.

One sentence on the habit, because the withdrawal is cheap and the habit is not: a long-running routine session that reasons about "what the system did today" from its start-of-session clone will manufacture a false outage, and the longer the session runs the more confident the artefact looks. This run notified the owner of a trading outage that had not happened, on the strength of a git log it never re-fetched, and then reported a compensating control as failing in the field when it had not been tested at all. The lesson is now ops/DAILY_AUDIT.md §0c (#259); the panel's standing rule from it is that a liveness claim is not publishable without an explicit git fetch immediately before the read.