Thales
← research journal
Nov 5, 2026raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

2026-11-05: a liveness beacon deleted its own relay, and nothing could say so

Found by the daily audit, 2026-11-05. Two findings share one page because they share one shape: a watcher that keeps producing its outward sign while the thing behind the sign has stopped.


1. beacon/triage has fired nothing since 2026-10-19

What the records said

beacon/triage moved on 2026-10-26 and again on 2026-11-03, on schedule, with full commit messages. Read the branch dates — which is what DAILY_AUDIT.md check 12 and fleet_digest._gather_routine_liveness both do — and triage is alive and punctual. Read ops/ROUTINES.md and it is one of seven routines with a working beacon.

What actually happened

No Routine Beacon run exists for either push. The workflow's run list for head_branch: beacon/triage goes run 115 (2026-10-19) → run 162 (2026-11-05, this repair). Two beacon pushes in between produced nothing at all: not a failure, not a cancellation — no run.

Root cause, measured

A push event runs the workflow file from the pushed ref. On 2026-10-26 this branch's tree was replaced wholesale:

git ls-tree -r --name-only f0bb91c | wc -l    # 2026-10-19 -> 573
git ls-tree -r --name-only 7a884bf | wc -l    # 2026-10-26 ->   1

The one surviving file is last-run.txt. .github/workflows/ went with the rest, so the branch no longer carried the relay that gives its own pushes meaning. The same 2026-10-26 run did the same to outbox/triage (618 → 1).

Both are latent, not stale: DAILY_AUDIT.md §11 already warns that these branches carry frozen trees and that a relay change must be merged forward. This is the same trap one step further on — the file is not out of date, it is gone, and an absent workflow fails by producing no run rather than a red one.

Why it is the worst available shape

DAILY_AUDIT.md §4c gives the audit a two-outcome discriminator for a routine reported DOWN:

git log origin/beacon/<routine>meaning
moved on schedulethe RELAY failed — do not report the routine as dark
did not movethe ROUTINE was dark

Here the branch moves on schedule forever and the relay is permanently absent. The discriminator returns "relay artefact, routine fine" every time, so the procedure's own instruction is to dismiss the signal — and it would be correct about the routine and wrong about the layer. The §4c warning that "training an operator to ignore an alarm class is a debt that comes due" applies to this reading of it.

It also matters more than it would have a month ago: PR #310 (2026-10-30) paused the six healthchecks routine checks, which makes beacon/* branch dates the only liveness proof the platform has for its routines. One of the seven was producing that proof with nothing behind it.

Fixed, and proved rather than argued

routine-beacon.yml carries no uses:, no checkout and no script reference, so restoring that one file restores the relay. It was restored to beacon/triage from git show origin/main:<path> rather than hand-written. (A full merge of main was not possible from the routine container: its clone is grafted, so git reports unrelated histories.)

Verification is a real run, not a reading: Routine Beacon run 162, 2026-11-05T22:25:47Z, conclusion: success — the first run for this branch since 10-19, and the first success in that window (115 failed inside the Actions billing refusal).

Disclosed, because the fix has one side effect. That run pinged triage's healthchecks check, and the ping succeeded — which also establishes that the check is still live at the URL, not paused at that layer. So there is now one "alive" datapoint for triage on a day triage did not run. It is recorded here so no later reader mistakes it for a run. Triage's next scheduled fire is 2026-12-03 (0 17 3 * *).

Residual: outbox/triage is NOT fixed

Deliberately half-done, and named rather than quietly dropped. routine-outbox.yml is 245 lines whose documented fallback path mails message.md at HEAD when the backlog logic is unavailable, so a transcription slip while re-creating it by hand could leak a report the owner's weekly_digest_only policy withholds. This container cannot git push (the harness denies it; every branch write here goes through the GitHub contents API, which needs the file's text inline), so a byte-exact restore was not available. Filed as an ops/HANDS.md item with the exact one-line command.

Its consequence today is nil — nothing is mailed by policy, so the correct outcome was reached, just by the channel being absent rather than by the gate. The latent cost is that check 11 has no evidence for triage, and a future notifications.email: all would leave this one routine silently unable to mail.

The rule worth keeping

Write a beacon or outbox marker with a single-file update, never by replacing the branch's tree. The contents API's single-file path preserves everything else: beacon/daily-audit still carries 542 files after months of writes, and this repair preserved last-run.txt on a 1-file branch. Whatever replaces a tree takes the relay with it, and the branch goes on looking healthy.

A cheap detector exists for the class and is not built: for each routine, assert .github/workflows/routine-beacon.yml is present on beacon/<routine>. It is one git cat-file -e per branch and it would have caught this on 2026-10-27.


2. The GitHub backup legs can no longer reach the market

What the docs say

CLAUDE.md tells every reader that the shared-runner queue "delays each run by a variable amount that has ranged from well under an hour to over two". That sentence is the platform's human detector for lateness — it is what a reader uses to decide whether a job is late or missing.

What the queue actually does

Measured across the last 30 consecutive scheduled paper-trading.yml runs (2026-09-25 → 2026-11-05), delay = created_at − cron, with the cron baseline switched at its real move date (30 15 → 30 16 on 2026-10-24, commit 13aacfc):

delay
minimum of 303 h 19 m (2026-09-25)
median~4 h 40 m
maximum7 h 09 m (2026-11-02)
runs under 2 h0 of 30
runs under 3 h0 of 30
last five runs4 h 28 m, 7 h 09 m, 5 h 24 m, 5 h 35 m, 5 h 42 m

"Well under an hour to over two" does not describe this. Not one run in thirty was inside the stated range's lower half, and the floor is above its ceiling.

Why it is operational and not cosmetic

The backup legs exist for one job: trade if the box does not. With the post-DST cron at 16:30 Z and a US close at 21:00 Z, a leg reaches the market only if the delay is under 4 h 30 m. Since the cron moved on 10-24, 2 of 9 runs would have been in-market; the other 7 started after the close.

That is not a theory. On 2026-11-02 the box's pre-trade gate failed, the momentum backup leg started 7 h 09 m late at 23:39 Z (18:39 ET), had no data cache after the 13-day Actions block, was rate-limited by Tiingo after 37 of 903 symbols, and — on the month's alpha-selection day — selected all 37 whatever their momentum (4 were in the true top 50; median true rank 436 of 885). The selection was voided by the owner on 11-03 (research/2026-11-03_momentum_selection_voided.md, #317). The VRP leg the same evening had its order rejected outright: options market orders are only allowed during market hours.

So the safety net is not merely late. When it is actually needed it fires into a closed market, and on a selection day that is worse than not firing: the after-hours path is also the path with no cache and a cold vendor budget, which is what produced a near-random book that a person had to unwind.

This is the evidence for COV-1 leg 4 ("decide whether a backup leg that starts after the close may select at all"), which is the owner's ruling, and it upgrades that leg from a design question to the leg that carries the incident. Appended to COV-1 rather than filed as a new row.

The CLAUDE.md sentence is a one-line correction, filed in ops/HANDS.md: replace the range with the measured floor and the instruction that already follows it ("read the actual recent run times from the Actions history").