# 2026-10-15 — the first day of a six-routine outage read as "7/7 reporting on schedule"

*Daily audit, window 2026-10-09 22:30Z → 2026-10-15 22:00Z. Companion to
`research/2026-10-15_every_weekday_routine_went_dark_and_the_alarm_went_to_the_dark_reader.md`,
which records the outage itself; this one records the detector defect the
outage exposed.*

## The observation

Six of the seven cloud routines stopped on **2026-10-12** and were still dark
through 10-15. The fleet digest's own dark-routine detector reported them
correctly on 10-13, 10-14 and 10-15 — it is a good detector. On **10-12 it
printed `routines: 7/7 reporting on schedule`.**

Reproduced against the real function, with the beacon branches untouched since
10-09, so the replay is faithful:

```
=== digest render date 2026-10-12   (its run was created 22:37:02Z)
  routines: 7/7 reporting on schedule
=== digest render date 2026-10-13
  ⚠ 5 routine(s) DARK — scheduled runs produced no beacon: ...
```

By 22:37Z on 10-12 the research panel (cron `0 13`), the weekly audit
(`0 14`) and queue triage (`0 16`) had each missed that day's slot by six to
nine hours. The digest said they were fine.

## Mechanism

`_gather_routine_liveness` (`fleet_digest.py`) counts cron fires over
`[last_beacon + 1 day, today - 1 day]`. Today's slot is excluded, and the
code says why:

> Yesterday is the last judgeable day; today's slot may not have arrived, and
> the digest never reads a clock to find out.

That is deliberate, and it is guarded: `_cron_fire_dates` is date-only by
contract, and `tests/test_clock_hermeticity.py` exists because a wall-clock
read inside a data filter fail-closed live trading on momentum and meanrev for
two days on 2026-07-17.

The 10-12 Monday is the worst case of the exclusion, because the window
`[10-10, 10-11]` is a weekend and contains no weekday fire at all. Every
routine scored `missed = 0`, and zero missed rendered as the all-clear.

## Why the fix is not "call them dark"

Measured from every beacon push on all seven beacon branches (subject-filtered
to real beacon commits), the lag between a routine's cron and its beacon:

| routine | cron | n | median | p90 | max |
|---|---|---|---|---|---|
| daily audit | 22:00Z | 36 | 21 m | 27 m | 37 m |
| research panel | 13:00Z | 36 | 53 m | 85 m | **582 m** |
| queue triage | 16:00Z | 14 | 22 m | 29 m | 50 m |
| research implementer | 16:00Z | 14 | 14 m | 402 m | **719 m** |
| weekly audit | 14:00Z | 14 | 22 m | 170 m | **839 m** |
| steward | 23:00Z | 2 | 11 m | 11 m | 11 m |

The tail is real and multi-hour. A same-day DARK claim therefore cannot be
made false-positive-free, and a false DARK is the expensive error here:
`ops/DAILY_AUDIT.md` §4c records that dismissing this alarm class was correct
on 09-21 and wrong on 09-22/09-23, by which point the reader had been trained
to dismiss it.

So the defect is not the missing alarm. **The defect is the affirmative green
printed over a slot nobody had checked** — the procedure's "a green you cannot
believe is worse than a red".

## The change (PR, observability tier)

1. A routine whose slot today has passed by **120 minutes** with no beacon is
   reported `⏳ awaiting today's beacon — slot passed, not yet proven`, and
   the all-clear line is withheld. The statement is true whichever way it
   resolves, and it **raises nothing**: `digest_exceptions` is untouched, so
   no new email and no new red run. 120 minutes clears every well-behaved p90
   above with margin, so a routine merely running long is never named.
2. The dark-routine exception names **every** dark routine. It was
   `dark[:4]`, so on 10-14 and 10-15 the annotation — the *only* record on a
   non-Friday — said "6 routine(s) dark" and named four, silently dropping the
   implementer and the weekly audit. Same shape as the 240-character
   `NEEDS-OWNER:` truncation #233 fixed the day before, one field over.

Replay with the fix, same real moments:

```
=== 10-12 22:37Z   ⏳ 3 routine(s) awaiting today's beacon — slot passed,
                      not yet proven: research panel, queue triage,
                      thales-weekly-audit
=== 10-14 19:44Z   ⚠ 5 routine(s) DARK ... plus
                   ⏳ 1 routine(s) awaiting: research implementer
```

The 10-14 line is the second gain: the implementer's Wednesday slot passed at
16:00Z and the old code could not say so until 10-15.

The clock read is threaded as `as_of` and fires only when `today` really is
today, so every fixed-date caller keeps its old answer. The first draft used
`now = now or datetime.now(...)` and the hermeticity tripwire rejected it by
name; the guard's own sanctioned vocabulary was adopted rather than widened.

## What is deliberately left open

Whether `awaiting` should itself become an exception — i.e. mail on a
non-Friday — is a change to what the owner receives, and the same judgement
DGX-1 already holds open for the unreadable-beacon case (`digest_exceptions`
does not treat `missed=None` as an exception either, pinned as a known
residual in `test_digest_cadence.py`). That is triage's call, or the owner's,
not this PR's. This PR only stops the detector asserting something it has not
checked.

Distinct from DGX-1, which is about the gather *failing* (`missed=None`); this
is about the gather *succeeding* with an incomplete window.