Thales
← research journal
Aug 6, 2026raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

2026-08-06 — the day the fleet flew blind, and nobody was told

Verdict: INCIDENT. All three sleeves skipped a trading day because GitHub gave the account no runners. Every alarm the repository owns was silenced by the same outage. The daily audit — running on Anthropic's infrastructure, the one layer outside GitHub's failure domain — found it ~5.5 hours later.

What happened

Every scheduled workflow fired on time and then died the same way:

workflowcreated (UTC)outcome
Paper Trading (momentum)16:21:35both jobs cancelled, no runner
Paper Trading (meanrev)16:22:39both jobs cancelled, no runner
Paper Trading (vrp)16:28:16both jobs cancelled, no runner
Fleet Digest17:25:20all six jobs cancelled, no runner
Skew Snapshot (19:00Z)never created
Heartbeat Monitor (20:00Z)never created

Each job reported conclusion: cancelled, runner_name: "", zero steps, and no logs, ~15 minutes after being queued. Steps are written by the runner as it executes them, so a job that never keeps one records none. No scheduled run was created at all after ~18:10Z, through at least 22:15Z. main carries no commit dated 2026-08-06.

The cause sits on GitHub's side — a capacity incident, or Actions blocked for the account (this is a private repository, so its minutes are metered). The audit runs in a sandbox that cannot reach githubstatus.com, so which one is not established here; it is the owner's first check.

Why it was silent

This is the second occurrence of the runner-loss class. The first (2026-07-24, vrp run 30108001231) produced the _alert.yml fix: the alert lives in a separate job, which GitHub dispatches to a fresh runner, so it survives the work runner's death. RUNBOOK.md named the residual that fix cannot cover, in the section that describes it:

The irreducible residual — a correlated outage where GitHub gives us no runners at all — is covered only by the healthchecks.io beacons above. That is why they matter more than they look.

2026-08-06 was that residual, exactly as written. The alert jobs needed the same runners the work jobs could not get, so all six of them were cancelled too. The prediction was correct and the mitigation held only as far as it claimed to.

Two layers remained, both outside GitHub:

  1. healthchecks.io beacons — they alarm on absence, from their own infrastructure, so a no-runner outage is precisely their case. Whether they fired tonight could not be checked from here (see below) and is the second thing for the owner to confirm.
  2. the cloud audit routines — which is what caught it.

The gap this leaves, and the fix

The fleet dead-man (heartbeat-monitor.yml) trips only after >4 calendar days of frozen equity — deliberately, so a weekend plus a holiday cannot false-alarm it. That tolerance is exactly wide enough to swallow a one-day total outage. Had the audit not run, nothing in the repository would ever have recorded that 2026-08-06 happened.

No in-band alarm can be present during an outage that takes out the whole band. So the fix is a deliberately late one: scripts/detect_runner_loss.py runs the next evening from the dead-man workflow, reads the Actions run history, and reds if any scheduled run in the trailing 26 hours shows the signature — cancelled + no runner + zero steps — reporting for each whether its alert job survived, i.e. whether anybody was actually paged. The owner learns the fleet flew blind one evening later, instead of never.

The failure mode of a retrospective detector is the opposite of silence: firing on ordinary cancels until it gets muted. Its negative controls are therefore the substance of its test file — an operator cancelling a manual dispatch, a timeout-minutes breach (also reported as cancelled, but with a runner and steps), a run-level cancel that stopped an already-started job, an ordinary red run, and an outage that has aged out of the window all stay quiet. An unreachable Actions API returns no verdict rather than an alarm: GitHub being slow is not evidence of an outage, and a detector that cries wolf gets muted — which is how this shop lost seven weeks once.

Replayed against the real run records of 2026-08-06, the detector reports all three affected runs and marks every one of them silent.

What it cost

  • momentum — no orders were due (selection is monthly; 08-06 was a vol-check day), so the cost is one missed daily kill-switch and vol check, and no equity mark. Drawdown stood at 8.1% against a 20% kill threshold with the switch inactive as of 08-05, so the missed check was very unlikely to have been the one that mattered.
  • vrp — a SPY put credit spread opened 08-03, expiring 08-31, went unmanaged for a day: the pinned open/manage/close rule could not evaluate. Defined-risk and 25 days from expiry, so the exposure is bounded, but this is the second time this sleeve has held an open short spread through a runner-loss day.
  • meanrev — one missed selection day on the never-promotable negative control. A record defect, not money.
  • captures — the 08-06 option-chain, skew, short-volume and borrow-flag snapshots do not exist and cannot be re-bought. One permanent notch in the asset the charter calls irreplaceable.
  • sitethales.report served 08-05 data all day; the export never ran.

Carried forward

The audit could not read the operator's inbox: this session's Gmail connector exposed no search, thread-read, or draft-create tool. So "what the alerts said" is unverified for this window — the half of the audit that diffs the inbox against the repository did not run. What the system did is established above; what the owner was told is not. Confirming whether the healthchecks beacons delivered is the one open question that decides whether 2026-08-06 was a caught outage or a completely unalarmed one.