2026-10-15 — the first day of a six-routine outage read as "7/7 reporting on schedule"
Daily audit, window 2026-10-09 22:30Z → 2026-10-15 22:00Z. Companion to
research/2026-10-15_every_weekday_routine_went_dark_and_the_alarm_went_to_the_dark_reader.md,
which records the outage itself; this one records the detector defect the
outage exposed.
The observation
Six of the seven cloud routines stopped on 2026-10-12 and were still dark
through 10-15. The fleet digest's own dark-routine detector reported them
correctly on 10-13, 10-14 and 10-15 — it is a good detector. On 10-12 it
printed routines: 7/7 reporting on schedule.
Reproduced against the real function, with the beacon branches untouched since 10-09, so the replay is faithful:
=== digest render date 2026-10-12 (its run was created 22:37:02Z)
routines: 7/7 reporting on schedule
=== digest render date 2026-10-13
⚠ 5 routine(s) DARK — scheduled runs produced no beacon: ...
By 22:37Z on 10-12 the research panel (cron 0 13), the weekly audit
(0 14) and queue triage (0 16) had each missed that day's slot by six to
nine hours. The digest said they were fine.
Mechanism
_gather_routine_liveness (fleet_digest.py) counts cron fires over
[last_beacon + 1 day, today - 1 day]. Today's slot is excluded, and the
code says why:
Yesterday is the last judgeable day; today's slot may not have arrived, and the digest never reads a clock to find out.
That is deliberate, and it is guarded: _cron_fire_dates is date-only by
contract, and tests/test_clock_hermeticity.py exists because a wall-clock
read inside a data filter fail-closed live trading on momentum and meanrev for
two days on 2026-07-17.
The 10-12 Monday is the worst case of the exclusion, because the window
[10-10, 10-11] is a weekend and contains no weekday fire at all. Every
routine scored missed = 0, and zero missed rendered as the all-clear.
Why the fix is not "call them dark"
Measured from every beacon push on all seven beacon branches (subject-filtered to real beacon commits), the lag between a routine's cron and its beacon:
| routine | cron | n | median | p90 | max |
|---|---|---|---|---|---|
| daily audit | 22:00Z | 36 | 21 m | 27 m | 37 m |
| research panel | 13:00Z | 36 | 53 m | 85 m | 582 m |
| queue triage | 16:00Z | 14 | 22 m | 29 m | 50 m |
| research implementer | 16:00Z | 14 | 14 m | 402 m | 719 m |
| weekly audit | 14:00Z | 14 | 22 m | 170 m | 839 m |
| steward | 23:00Z | 2 | 11 m | 11 m | 11 m |
The tail is real and multi-hour. A same-day DARK claim therefore cannot be
made false-positive-free, and a false DARK is the expensive error here:
ops/DAILY_AUDIT.md §4c records that dismissing this alarm class was correct
on 09-21 and wrong on 09-22/09-23, by which point the reader had been trained
to dismiss it.
So the defect is not the missing alarm. The defect is the affirmative green printed over a slot nobody had checked — the procedure's "a green you cannot believe is worse than a red".
The change (PR, observability tier)
- A routine whose slot today has passed by 120 minutes with no beacon is
reported
⏳ awaiting today's beacon — slot passed, not yet proven, and the all-clear line is withheld. The statement is true whichever way it resolves, and it raises nothing:digest_exceptionsis untouched, so no new email and no new red run. 120 minutes clears every well-behaved p90 above with margin, so a routine merely running long is never named. - The dark-routine exception names every dark routine. It was
dark[:4], so on 10-14 and 10-15 the annotation — the only record on a non-Friday — said "6 routine(s) dark" and named four, silently dropping the implementer and the weekly audit. Same shape as the 240-characterNEEDS-OWNER:truncation #233 fixed the day before, one field over.
Replay with the fix, same real moments:
=== 10-12 22:37Z ⏳ 3 routine(s) awaiting today's beacon — slot passed,
not yet proven: research panel, queue triage,
thales-weekly-audit
=== 10-14 19:44Z ⚠ 5 routine(s) DARK ... plus
⏳ 1 routine(s) awaiting: research implementer
The 10-14 line is the second gain: the implementer's Wednesday slot passed at 16:00Z and the old code could not say so until 10-15.
The clock read is threaded as as_of and fires only when today really is
today, so every fixed-date caller keeps its old answer. The first draft used
now = now or datetime.now(...) and the hermeticity tripwire rejected it by
name; the guard's own sanctioned vocabulary was adopted rather than widened.
What is deliberately left open
Whether awaiting should itself become an exception — i.e. mail on a
non-Friday — is a change to what the owner receives, and the same judgement
DGX-1 already holds open for the unreadable-beacon case (digest_exceptions
does not treat missed=None as an exception either, pinned as a known
residual in test_digest_cadence.py). That is triage's call, or the owner's,
not this PR's. This PR only stops the detector asserting something it has not
checked.
Distinct from DGX-1, which is about the gather failing (missed=None); this
is about the gather succeeding with an incomplete window.