2026-11-05: a liveness beacon deleted its own relay, and nothing could say so
Found by the daily audit, 2026-11-05. Two findings share one page because they share one shape: a watcher that keeps producing its outward sign while the thing behind the sign has stopped.
1. beacon/triage has fired nothing since 2026-10-19
What the records said
beacon/triage moved on 2026-10-26 and again on 2026-11-03, on schedule, with
full commit messages. Read the branch dates — which is what DAILY_AUDIT.md
check 12 and fleet_digest._gather_routine_liveness both do — and triage is
alive and punctual. Read ops/ROUTINES.md and it is one of seven routines with
a working beacon.
What actually happened
No Routine Beacon run exists for either push. The workflow's run list for
head_branch: beacon/triage goes run 115 (2026-10-19) → run 162
(2026-11-05, this repair). Two beacon pushes in between produced nothing at
all: not a failure, not a cancellation — no run.
Root cause, measured
A push event runs the workflow file from the pushed ref. On 2026-10-26
this branch's tree was replaced wholesale:
git ls-tree -r --name-only f0bb91c | wc -l # 2026-10-19 -> 573
git ls-tree -r --name-only 7a884bf | wc -l # 2026-10-26 -> 1
The one surviving file is last-run.txt. .github/workflows/ went with the
rest, so the branch no longer carried the relay that gives its own pushes
meaning. The same 2026-10-26 run did the same to outbox/triage (618 → 1).
Both are latent, not stale: DAILY_AUDIT.md §11 already warns that these
branches carry frozen trees and that a relay change must be merged forward.
This is the same trap one step further on — the file is not out of date, it is
gone, and an absent workflow fails by producing no run rather than a red one.
Why it is the worst available shape
DAILY_AUDIT.md §4c gives the audit a two-outcome discriminator for a routine
reported DOWN:
git log origin/beacon/<routine> | meaning |
|---|---|
| moved on schedule | the RELAY failed — do not report the routine as dark |
| did not move | the ROUTINE was dark |
Here the branch moves on schedule forever and the relay is permanently absent. The discriminator returns "relay artefact, routine fine" every time, so the procedure's own instruction is to dismiss the signal — and it would be correct about the routine and wrong about the layer. The §4c warning that "training an operator to ignore an alarm class is a debt that comes due" applies to this reading of it.
It also matters more than it would have a month ago: PR #310 (2026-10-30) paused
the six healthchecks routine checks, which makes beacon/* branch dates the
only liveness proof the platform has for its routines. One of the seven was
producing that proof with nothing behind it.
Fixed, and proved rather than argued
routine-beacon.yml carries no uses:, no checkout and no script reference, so
restoring that one file restores the relay. It was restored to beacon/triage
from git show origin/main:<path> rather than hand-written. (A full merge of
main was not possible from the routine container: its clone is grafted, so git
reports unrelated histories.)
Verification is a real run, not a reading: Routine Beacon run 162,
2026-11-05T22:25:47Z, conclusion: success — the first run for this branch
since 10-19, and the first success in that window (115 failed inside the
Actions billing refusal).
Disclosed, because the fix has one side effect. That run pinged triage's
healthchecks check, and the ping succeeded — which also establishes that the
check is still live at the URL, not paused at that layer. So there is now one
"alive" datapoint for triage on a day triage did not run. It is recorded here so
no later reader mistakes it for a run. Triage's next scheduled fire is
2026-12-03 (0 17 3 * *).
Residual: outbox/triage is NOT fixed
Deliberately half-done, and named rather than quietly dropped. routine-outbox.yml
is 245 lines whose documented fallback path mails message.md at HEAD when
the backlog logic is unavailable, so a transcription slip while re-creating it by
hand could leak a report the owner's weekly_digest_only policy withholds. This
container cannot git push (the harness denies it; every branch write here goes
through the GitHub contents API, which needs the file's text inline), so a
byte-exact restore was not available. Filed as an ops/HANDS.md item with the
exact one-line command.
Its consequence today is nil — nothing is mailed by policy, so the correct
outcome was reached, just by the channel being absent rather than by the gate.
The latent cost is that check 11 has no evidence for triage, and a future
notifications.email: all would leave this one routine silently unable to mail.
The rule worth keeping
Write a beacon or outbox marker with a single-file update, never by replacing
the branch's tree. The contents API's single-file path preserves everything
else: beacon/daily-audit still carries 542 files after months of writes, and
this repair preserved last-run.txt on a 1-file branch. Whatever replaces a
tree takes the relay with it, and the branch goes on looking healthy.
A cheap detector exists for the class and is not built: for each routine, assert
.github/workflows/routine-beacon.yml is present on beacon/<routine>. It is
one git cat-file -e per branch and it would have caught this on 2026-10-27.
2. The GitHub backup legs can no longer reach the market
What the docs say
CLAUDE.md tells every reader that the shared-runner queue "delays each run by
a variable amount that has ranged from well under an hour to over two". That
sentence is the platform's human detector for lateness — it is what a reader
uses to decide whether a job is late or missing.
What the queue actually does
Measured across the last 30 consecutive scheduled paper-trading.yml runs
(2026-09-25 → 2026-11-05), delay = created_at − cron, with the cron baseline
switched at its real move date (30 15 → 30 16 on 2026-10-24, commit 13aacfc):
| delay | |
|---|---|
| minimum of 30 | 3 h 19 m (2026-09-25) |
| median | ~4 h 40 m |
| maximum | 7 h 09 m (2026-11-02) |
| runs under 2 h | 0 of 30 |
| runs under 3 h | 0 of 30 |
| last five runs | 4 h 28 m, 7 h 09 m, 5 h 24 m, 5 h 35 m, 5 h 42 m |
"Well under an hour to over two" does not describe this. Not one run in thirty was inside the stated range's lower half, and the floor is above its ceiling.
Why it is operational and not cosmetic
The backup legs exist for one job: trade if the box does not. With the post-DST cron at 16:30 Z and a US close at 21:00 Z, a leg reaches the market only if the delay is under 4 h 30 m. Since the cron moved on 10-24, 2 of 9 runs would have been in-market; the other 7 started after the close.
That is not a theory. On 2026-11-02 the box's pre-trade gate failed, the
momentum backup leg started 7 h 09 m late at 23:39 Z (18:39 ET), had no data
cache after the 13-day Actions block, was rate-limited by Tiingo after 37 of
903 symbols, and — on the month's alpha-selection day — selected all 37
whatever their momentum (4 were in the true top 50; median true rank 436 of
885). The selection was voided by the owner on 11-03
(research/2026-11-03_momentum_selection_voided.md, #317). The VRP leg the same
evening had its order rejected outright: options market orders are only allowed during market hours.
So the safety net is not merely late. When it is actually needed it fires into a closed market, and on a selection day that is worse than not firing: the after-hours path is also the path with no cache and a cold vendor budget, which is what produced a near-random book that a person had to unwind.
This is the evidence for COV-1 leg 4 ("decide whether a backup leg that starts after the close may select at all"), which is the owner's ruling, and it upgrades that leg from a design question to the leg that carries the incident. Appended to COV-1 rather than filed as a new row.
The CLAUDE.md sentence is a one-line correction, filed in ops/HANDS.md:
replace the range with the measured floor and the instruction that already
follows it ("read the actual recent run times from the Actions history").