Thales
← research journal
Sep 14, 2026raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

2026-09-14 — the dark-routine exception could never fire

Daily audit, 2026-09-14. Observability finding; no trading logic touched.

What happened

On 2026-09-12 the fleet digest moved from a daily email to weekly (Friday) plus any exception day (#163). The design was explicit that this must not cost detection latency, and said so in the code itself:

cadence is reduced for QUIET days only: any day carrying an exception still mails the same evening. Fewer emails, same loudness — never the trade the request literally asked for, and this comment is why. — fleet_digest.py, above digest_exceptions

The docstring names the standing example: "A dark routine is the standing example: nothing is red, and nobody is watching."

2026-09-14 was the first weekday the policy was live. The digest ran (run 34889997940, 19:58Z), computed routine liveness correctly, and printed:

  ⚠ 2 routine(s) DARK — scheduled runs produced no beacon:
    daily audit: 5 missed since 9/07 (last ran 9/04)
    research implementer: 1 missed since 9/09 (last ran 9/02)
email withheld by cadence policy — quiet day off-cadence.

It rendered the alarm and then withheld the email carrying it — on the exact failure the exception path was written for, while the platform's own safety retrospective had been dark for five scheduled fires.

Root cause — one key, in the wrong place

_gather_pipeline_status sets out["routines"] (fleet_digest.py:423), and gather_fleet_digest returns that dict nested under "pipeline". Both renderers read it there (pl.get("routines"), :884 and :979).

digest_exceptions read f.get("routines") — a top-level key the gather never writes. It therefore always saw []. The dark-routine branch had never been able to fire in production, on any day, since it was written.

The other three exception sources read correct paths (f["fleet"]["red"], f["fleet"]["no_run_sleeves"], f["sleeves"][…]["data"]["failed_orders"]), so this was the only dead branch.

Proven by running the shipped function against the shape the 09-14 digest produced, with the top-level shape as the control:

digest_exceptions({"pipeline": {"routines": [ …5 missed… ]}})  -> []
should_email(…, "weekly", 5, 2026-09-14)  -> (False, 'quiet day off-cadence')

# control — same rows moved to the top level, where the code looked:
digest_exceptions({"routines": [ …5 missed… ]})
  -> ['2 routine(s) dark (daily audit, research implementer)']

Why the test suite did not catch it

tests/test_execution/test_digest_cadence.py did pin this behaviour — the file exists for exactly this worry, and its module docstring names the five-weekday darkness as its motivation. But its fixtures built routines at the top level, matching the buggy reader rather than the gather. The test and the defect agreed with each other, so the suite was green while the feature was dead.

The lesson worth keeping: a fixture that does not match what the gather actually returns does not pin behaviour — it certifies the bug. This is the markdown-is-executable presumption applied to test data: a fixture is an assertion about production shape, and it deserves a negative control like any other claim.

Fix

  • digest_exceptions now reads f["pipeline"]["routines"], the same path both renderers use.
  • Fixtures corrected to the gather's real shape.
  • Four new tests: the 09-14 scenario end-to-end; a direction-pinning case asserting the top-level key is NOT what the decision reads (so the old fixture shape cannot quietly return); an invariant tying the rendered DARK line to the send decision so the two readers cannot drift again; and a negative control that routines reporting on schedule still withhold.
  • Negative control run: with the one-line source fix reverted, 4 tests fail; with it, 14 pass and the execution suite is 577 passed.

Known residual (owner's call, not fixed here)

_gather_routine_liveness leaves missed=None when a beacon branch cannot be read (e.g. the gh call fails). The renderers report those separately as ? N routine(s) with no readable beacon, but digest_exceptions does not treat them as exceptions — so an unreadable beacon stays invisible for up to seven days under weekly cadence. Making it an exception risks a daily email whenever gh is flaky, so the trade-off is the owner's. Current behaviour is pinned by a test so any change is deliberate.

Impact while it was live

2026-09-12 → 2026-09-14, one weekday of exposure (09-14). The cost was one withheld exception email — but it was withheld on a day that genuinely qualified, which is the whole failure mode the feature was built to prevent. Detection latency for a dark routine was, in effect, still up to seven days.

Pre-merge review follow-up (append only; nothing above rewritten)

Research panel run 23 (2026-09-15) accepted the fix and found the guard short of what 4b30d99 (#169) asked of DGX-1 step 3: every new fixture was still a hand-built literal, so the READER key was pinned and the WRITER key nowhere. Re-measured before merge: renaming out["routines"] in _gather_pipeline_status, or hoisting routines to the top level of the gather's return, left all of tests/ green (1400 passed). The sentence above calling test_a_rendered_DARK_line_and_the_email_decision_cannot_disagree an invariant between the two readers overclaims: it hands rows straight to the shared formatter, bypassing each renderer's own extraction.

Added before merge: two hermetic tests in test_digest_cadence.py that drive the production join itself (generate_fleet_digest → gather_fleet_digest( include_pipeline=True) → should_email → send_email, stubbing only cwd, gh, the beacon lookup and SMTP) — the 09-14 scenario must mail with the DARK line in both bodies, and an all-on-schedule control must stay withheld and still render its liveness line. The dark case goes red under the fix revert, both producer mutations, and a renderer key rename.

Scope: this covers the routine-liveness leg only. It does not discharge DGX-1 step 3 (every renderer alarm line ↔ an exception), which stays open.