# The watcher of the watchers went blind, and said so in the quietest possible way **2026-10-23, daily audit.** Two defects in the routine-liveness detector, found by reading the Friday fleet digest — the only email the owner receives since 2026-10-07 — against the repository's own beacon branches. ## 1. All seven beacons unreadable on the box, silently The digest's RESEARCH PIPELINE section ended with: > `? 7 routine(s) with no readable beacon: daily audit, steward, research > panel, queue triage, research implementer, research audit, > thales-weekly-audit` Each cloud routine proves it ran by pushing a marker commit to `beacon/`; the digest reads those branches and names the routines that have stopped. On 2026-10-23 it could read none of them. Three consequences, the third being the one that bites: 1. the DARK branch of `digest_exceptions` only fires on a *readable* stale beacon, so it was skipped rather than triggered; 2. `steward_dark` consumes the same verdict and was disarmed with it; 3. **this is the failure the fallback was written to prevent.** The docstring of `_last_beacon_date_git`, added 2026-10-19 when the digest moved to tbx-win, says verbatim: *"Without it every routine reads `unknown` there — and the steward-dark alarm, which needs a DARK verdict, could never fire."* And it was not hypothetical: `beacon/panel` went 2026-10-21 → 2026-10-23, skipping 10-22. The research panel was genuinely dark the day before, and the email named no dark routine. ### Detector, not system — established by execution Both code paths were run in a cloud container against the real branches, as of 2026-10-23 15:30Z, and both returned identical dates for all seven: ``` daily audit 10-22 · steward 10-22 · panel 10-23 · triage 10-19 implementer 10-21 · research-audit 10-01 · weekly-audit 10-19 → "routines: 7/7 reporting on schedule" ``` So the branches are readable, the dates are present and the logic is right. The failure is host-specific to the box. ### Cause: strongly indicated, not established The digest shells out to exactly three external commands, **all by bare name**: `gh api` (beacon, first choice), `git fetch`/`git log` (beacon fallback), and `gh pr list`. `gh` being unusable on the box is proved by the digest's own output — the same email printed *"open PRs: unavailable (gh not usable in this run)"*. For `git` the evidence is circumstantial but pointed: the box's own session script does not trust a bare `git` either, hardcoding the absolute path at `scripts/win/session.py:46`: ```python GIT = r"C:\Program Files\Git\cmd\git.exe" if os.name == "nt" else "git" ``` A child process that does not inherit a usable PATH is the leading explanation. It is **not** called established: credentials on the beacon refspec, git's "dubious ownership" refusal under the box's `thales` account, and a fetch timeout all produce the identical symptom — because the function discarded the reason. ### The defect that needed no access to the box **The function threw the diagnosis away.** On every failure path it returned `None` with no log line, no exit code and no stderr. A host-specific breakage of the whole alarm layer surfaced as one `?` in an email, and the cause had to be inferred from a hardcoded path in an unrelated script. This is the same shape as the previous evening's finding — a detector that cannot say why it failed — one layer up. Fixed: tools are resolved via `shutil.which` with a `THALES_GIT` / `THALES_GH` override (the one remedy that needs no access to the box), every failure path logs its own cause at WARNING, and a wholly unreadable beacon layer is raised as a digest exception. ### The exception reverses an explicitly owner-deferred call `test_unreadable_beacon_is_NOT_an_exception_today_known_residual` pinned the old silence and said: *"Making it an exception risks a daily email on a flaky `gh`, so the call is the owner's."* All three premises have since changed: 1. **"a daily email"** — since 2026-10-07 the owner receives one email a week; a non-Friday exception is recorded for the steward, not mailed. 2. **"a flaky `gh`"** — since 2026-10-19 a plain-git fallback exists, so `gh` alone can no longer make a beacon unreadable. Reaching this state needs both paths to fail. 3. it **stopped being hypothetical**, in the owner's only email, on a day a routine was dark. The new condition is also strictly narrower than the deferred one: it fires on ALL beacons unreadable (the instrument down), never on one (a newly registered routine with no branch yet), which is controlled by a retained negative test. The reversal is proposed, not assumed — flagged `NEEDS-OWNER` because the pin named the owner as the decider. ## 2. The dark-routine count has no memory, so an outage that ENDS is invisible Independent of the above, and found while asking why yesterday's dark panel was not named today. The missed-run count is measured from the **last** beacon forward: ```python fires = _cron_fire_dates(cron, last + timedelta(days=1), today - timedelta(days=1)) row["missed"] = len(fires) ``` The window *starts at the most recent beacon*. The panel's beacon moved again at 13:33Z on 10-23, setting `last = 2026-10-23`, emptying the window and giving `missed = 0` — measured, the day after it was dark. **One recovery erases the entire record of the outage.** The detector reports only *currently ongoing* darkness. Before 2026-10-07 this cost little, because a digest arrived every weekday and an ongoing outage was caught the next morning. Under a weekly digest the two compose badly: > A routine can be dark Monday through Thursday, run once on Friday morning > before the 15:30Z gather, and the Friday email will read > **"routines: 7/7 reporting on schedule."** Four missed runs in five, reported as perfect. It is the mechanism that hid one dark day this week, scaled by the weekly cadence. **Deliberately not fixed in the same change.** The honest fix alters what `missed` means, and `missed` feeds `steward_dark` and was calibrated on 2026-10-15 after a false-alarm episode. It needs its own change with its own negative controls — most likely reading the beacon's full commit-date set and counting fire dates with no commit, reported as a field additional to `missed` rather than a redefinition of it. Recorded here so the gap is not carried silently. ## The lesson Both halves are the same lesson in different registers, and it is the standing one: *a green you cannot believe is worse than a red.* The first defect produced a `?` where an alarm belonged; the second produces a confident green over a real outage. Neither was visible from the inbox and neither was visible from the repository — only from holding the two side by side, which is the whole method. A third thing worth keeping: the prediction in the previous evening's report, *"today's miss becomes a named DARK routine in tomorrow's Friday email"*, was wrong — and it was wrong for **two independent reasons**, either of which alone would have sufficed. A forecast that fails is the cheapest detector audit available, and it is worth writing predictions down precisely so they can fail legibly.