2026-08-06 — the day the fleet flew blind, and nobody was told
Verdict: INCIDENT. All three sleeves skipped a trading day because GitHub gave the account no runners. Every alarm the repository owns was silenced by the same outage. The daily audit — running on Anthropic's infrastructure, the one layer outside GitHub's failure domain — found it ~5.5 hours later.
What happened
Every scheduled workflow fired on time and then died the same way:
| workflow | created (UTC) | outcome |
|---|---|---|
| Paper Trading (momentum) | 16:21:35 | both jobs cancelled, no runner |
| Paper Trading (meanrev) | 16:22:39 | both jobs cancelled, no runner |
| Paper Trading (vrp) | 16:28:16 | both jobs cancelled, no runner |
| Fleet Digest | 17:25:20 | all six jobs cancelled, no runner |
| Skew Snapshot (19:00Z) | — | never created |
| Heartbeat Monitor (20:00Z) | — | never created |
Each job reported conclusion: cancelled, runner_name: "", zero steps, and
no logs, ~15 minutes after being queued. Steps are written by the runner as it
executes them, so a job that never keeps one records none. No scheduled run was
created at all after ~18:10Z, through at least 22:15Z. main carries no commit
dated 2026-08-06.
The cause sits on GitHub's side — a capacity incident, or Actions blocked for
the account (this is a private repository, so its minutes are metered). The
audit runs in a sandbox that cannot reach githubstatus.com, so which one is
not established here; it is the owner's first check.
Why it was silent
This is the second occurrence of the runner-loss class. The first
(2026-07-24, vrp run 30108001231) produced the _alert.yml fix: the alert
lives in a separate job, which GitHub dispatches to a fresh runner, so
it survives the work runner's death. RUNBOOK.md named the residual that fix
cannot cover, in the section that describes it:
The irreducible residual — a correlated outage where GitHub gives us no runners at all — is covered only by the healthchecks.io beacons above. That is why they matter more than they look.
2026-08-06 was that residual, exactly as written. The alert jobs needed the same runners the work jobs could not get, so all six of them were cancelled too. The prediction was correct and the mitigation held only as far as it claimed to.
Two layers remained, both outside GitHub:
- healthchecks.io beacons — they alarm on absence, from their own infrastructure, so a no-runner outage is precisely their case. Whether they fired tonight could not be checked from here (see below) and is the second thing for the owner to confirm.
- the cloud audit routines — which is what caught it.
The gap this leaves, and the fix
The fleet dead-man (heartbeat-monitor.yml) trips only after >4 calendar
days of frozen equity — deliberately, so a weekend plus a holiday cannot
false-alarm it. That tolerance is exactly wide enough to swallow a one-day
total outage. Had the audit not run, nothing in the repository would ever have
recorded that 2026-08-06 happened.
No in-band alarm can be present during an outage that takes out the whole
band. So the fix is a deliberately late one: scripts/detect_runner_loss.py
runs the next evening from the dead-man workflow, reads the Actions run
history, and reds if any scheduled run in the trailing 26 hours shows the
signature — cancelled + no runner + zero steps — reporting for each whether
its alert job survived, i.e. whether anybody was actually paged. The owner
learns the fleet flew blind one evening later, instead of never.
The failure mode of a retrospective detector is the opposite of silence: firing
on ordinary cancels until it gets muted. Its negative controls are therefore
the substance of its test file — an operator cancelling a manual dispatch, a
timeout-minutes breach (also reported as cancelled, but with a runner and
steps), a run-level cancel that stopped an already-started job, an ordinary red
run, and an outage that has aged out of the window all stay quiet. An
unreachable Actions API returns no verdict rather than an alarm: GitHub being
slow is not evidence of an outage, and a detector that cries wolf gets muted —
which is how this shop lost seven weeks once.
Replayed against the real run records of 2026-08-06, the detector reports all three affected runs and marks every one of them silent.
What it cost
- momentum — no orders were due (selection is monthly; 08-06 was a vol-check day), so the cost is one missed daily kill-switch and vol check, and no equity mark. Drawdown stood at 8.1% against a 20% kill threshold with the switch inactive as of 08-05, so the missed check was very unlikely to have been the one that mattered.
- vrp — a SPY put credit spread opened 08-03, expiring 08-31, went unmanaged for a day: the pinned open/manage/close rule could not evaluate. Defined-risk and 25 days from expiry, so the exposure is bounded, but this is the second time this sleeve has held an open short spread through a runner-loss day.
- meanrev — one missed selection day on the never-promotable negative control. A record defect, not money.
- captures — the 08-06 option-chain, skew, short-volume and borrow-flag snapshots do not exist and cannot be re-bought. One permanent notch in the asset the charter calls irreplaceable.
- site —
thales.reportserved 08-05 data all day; the export never ran.
Carried forward
The audit could not read the operator's inbox: this session's Gmail connector exposed no search, thread-read, or draft-create tool. So "what the alerts said" is unverified for this window — the half of the audit that diffs the inbox against the repository did not run. What the system did is established above; what the owner was told is not. Confirming whether the healthchecks beacons delivered is the one open question that decides whether 2026-08-06 was a caught outage or a completely unalarmed one.