Thales
← research journal
raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

DGX-1 — the new weekly-email gate is blind to alarms the digest's own renderer prints: a failed VRP close, a corrupt option-position file, and "no readable beacon" all read as a quiet day

status: open · raised panel run 22 (2026-09-14, change-review catch on PR #163 — does NOT count toward the frontier cadence) · class: loudness (G2 time-to-loud) on the operator's ONLY push channel · effort ~0.5 pd · horizon: live from today — 2026-09-14 is the first weekday under email_policy: weekly

Plain-language summary for an owner reading one paragraph. Saturday's change did the right thing with a request that could have gone badly: asked for "one email a week instead of five", it cut the cadence for QUIET days only and kept every exception day mailing the same evening. The exception list it built is good. The problem is what feeds it. digest_exceptions() asks four questions, and for the VRP sleeve — the one live options sleeve — two of them cannot be answered, because the option sleeve's gather never produces the fields they read. The result: the digest can render 🔴 POSITION FILE CORRUPT or carry a failed close in its state, and the gate still returns "quiet day off-cadence" and withholds the mail until Friday. A third case is the routine-liveness probe: when it cannot read the beacons at all, it degrades to "unknown", and unknown is not an exception — so the dark-routine trigger goes silent exactly when its own data source breaks. The defect is a shape, not three bugs: the renderer and the gate were written from different sources of truth, and only the renderer knows what counts as red.

Mechanism — reproduced against the real functions at HEAD 18a510b

Every case below was executed, not read.

(1) A failed VRP order can never be an exception. digest_exceptions (fleet_digest.py:1157, the per-sleeve loop at :1181-1187) reads d["failed_orders"] / d["failed_standing"] per sleeve. Those keys are produced by digest.gather_digest (digest.py:247-249), which serves the weights sleeves. The VRP sleeve is gathered by _gather_option_sleeve (fleet_digest.py:68-172) — a different function whose return dict contains neither key. So n_failed and n_standing are both 0, 0 > 0 is False, and no option-sleeve failure of any age can make a day an exception. Executed:

vrp data = _gather_option_sleeve's exact key set, orders_today carrying
           {'status': 'failed', 'cause': 'vrp_close'}
digest_exceptions(f)             -> []
should_email(f,'weekly',5, Tue)  -> (False, 'quiet day off-cadence')

The body is silent too: _text_option_section (:799-827) and _html_option_section (:1053) render no failed-order line at all, so both failed_orders reads in the module (:781, :1034) are inside the weights sections. Nothing in the fleet email has ever shown a failed VRP order. Before #163 that was a rendering gap; #163 made it load-bearing.

Is it backstopped? Partly, and the gap is the standing case. A fresh failed close does red the run: option_intents.py:76-79 stamps action: "close", is_de_risking (failed_streaks.py:27-28) accepts action == "close", so cli.py:1176 raises Exit(4) and paper-trading-vrp.yml's _alert job fires. But from the second session onward the same symbol is in prev_failed — and note the symbol an option order carries is the underlying, "SPY" (option_intents.py:78), so the masking is coarser still — fresh is empty, the run exits 0 by FOS-1's deliberate design — and the ruling's stated reason is that "a standing streak is the digest's job — its streak line names it". For the option sleeve there is no such line and now no such email. A VRP close that keeps failing leaves a live short spread the book cannot exit, with green CI and no mail until Friday.

Why the digest specifically, for this sleeve. RUNBOOK.md:190-198 records incident 2026-07-24: GitHub reclaimed the VRP trade job's runner mid-run, so the workflow's own alert step could not execute — "the sleeve skipped a trading day holding an open short put spread and nothing paged; only the fleet digest noticed." That is the shop's own written finding that the digest is VRP's last-resort detector when the in-job path dies. The no_run half of that incident is covered by digest_exceptions. The failed-order half is the hole this row names.

Honest severity: latent, not active. Measured: data/state/vrp/ holds zero failed-order records across the sleeve's whole live history since 2026-07-13. This hole has never been exercised. It is filed as a loudness defect found before its first incident — which is the only time such a thing is cheap — not as a live outage. The contrast is meanrev's AVB, still failing its emergency exit on 2026-09-11, on the covered (weights) path.

(2) A corrupt VRP position file renders 🔴 and the gate calls it quiet. _gather_option_sleeve:160 emits corrupt_position; _text_option_section:805-806 prints 🔴 POSITION FILE CORRUPT — manual reconciliation required. digest_exceptions never reads the key, and any_red (:496-505) does not include it either. Executed: body prints the 🔴 line, gate returns (False, 'quiet day off-cadence'). Backstopped, and honestly so: vrp_daily._read_position (:82-93) raises RuntimeError on corruption, so the trading run itself reds and its own alert job fires. This instance is the cheapest proof of the shape, not the dangerous one — keep it as a test case, not as the argument.

This one was already known, at LOW, and #163 silently promoted it. TECH_DEBT.md:687-689 carries: "VRP corrupt-position renders a red section but is absent from the fleet red/subject rollup (LOW) — a corrupt vrp position wouldn't red the subject." That was accurate and fairly rated when the email went out every weekday: the cost was a subject line that under-sold its own body. Saturday's change re-used the same rollup as the send/withhold decision, so the cost is now the body not arriving for up to four days. Nobody did anything wrong — the LOW rating was correct for the world it was written in. This is the "a TECH_DEBT item that later work silently made urgent" case, and it is the single clearest argument for the structural fix in step 3 below: the next feature to read that rollup will inherit the same blind spot again.

(3) The dark-routine trigger fails quiet when the beacon probe fails. _gather_routine_liveness:311-325 sets missed = None whenever _last_beacon_date (:275-292, a gh api shell-out) returns None — which is every routine at once if gh fails (no token, rate limit, API outage) or a beacon branch is gone. _routine_liveness_lines:333-347 renders those as ? N routine(s) with no readable beacon; digest_exceptions:1174-1178 filters on r.get("missed"), and None is falsy. Executed with all six routines unreadable: body prints the ? line, digest_exceptions returns [], gate returns quiet. This one is latent, not active — the workflow does set GH_TOKEN with contents: read (fleet-digest.yml:88-89, :37), so the probe works today. It is the fail-quiet direction on a notification gate, which is the thing this shop has decided twice already it does not accept (exit 3 for the dead SMTP path; --fail-on-no-run for the rendered-only DID-NOT-RUN signal).

Why the new tests do not catch any of it

tests/test_execution/test_digest_cadence.py (9 tests, all green on this checkout — run, not assumed) is a good test file: it has a real negative control (the same quiet Tuesday sends under always), and test_a_standing_failure_alone_is_not_a_new_exception pins the AVB case the feature was built around. Its blind spot is structural, not sloppy — every payload is hand-built:

(_with(sleeves=[{"name": "meanrev", "ok": True, "data": {
    "failed_orders": [{"symbol": "AVB"}, {"symbol": "XYZ"}],
    "failed_standing": [{"symbol": "AVB"}]}}]), "1 NEW failed order"),

The fixture supplies exactly the two keys the code reads, so the tests verify the logic given the right shape and can never discover that one of the two real gathers does not produce that shape. No test in the file constructs a payload from _gather_option_sleeve, and none asserts anything about the option sleeve at all. A detector whose fixture is written from the consumer's expectations rather than the producer's output cannot fail in this direction — which is why step 3 below is the part worth building.

Prior art, checked before filing

  • research/queue/ (all four dirs) and the frozen research/RESEARCH_QUEUE.md: no dgx-* slug, and no row on digest_exceptions / should_email / the cadence policy — the feature is two days old.
  • TECH_DEBT.md: one hit, instance (2), quoted above at LOW; no entry for instance (1) or (3).
  • research/2026-08-01_quant_panel_suggestions.md reject ledger: no hit.
  • web/public/data/killlist.json: strategy kills only; not applicable.
  • Nearest neighbours are FOS-2 and FOS-3, discussed next. Neither covers this.

Why this is not FOS-2, and how they relate

FOS-2 is the weights renderer printing a flat failed-order count with no NEW/STANDING split. DGX-1 is the option sleeve having no failed-order path at all — gather, renderer, or gate — plus the gate/renderer divergence in (2) and (3). They share a diagnosis (a reporting feature built in one renderer and not the other, twice now: see the 2026-07-20 scar quoted at fleet_digest.py:777-779) and should be triaged together; the fix shapes compose. Neither subsumes the other.

Fix shape (propose-only — the panel does not implement)

One predicate, both consumers. The gate should not re-derive what is alarming; it should ask the same code the renderer asks. Concretely:

  1. Give _gather_option_sleeve the two keys the contract already implies — failed_orders (it already holds orders_today with status) and failed_standing (the same failed_streaks split the weights gather runs, which works on OCC symbols unchanged). That closes (1) at the source and gives the option section a failed-order line for free.
  2. Add corrupt_position and the missed is None ("liveness unknown") case to the exception set.
  3. The structural half: a test that walks the alarm conditions the renderers can emit (🔴, !!, the ?-liveness line) and asserts each one produces a non-empty digest_exceptions. A new red line added to a renderer without a matching exception must fail the suite — that is the detector that makes this class non-recurring, and it is the reason to treat this as one item rather than three patches.

Pre-registered kill criterion

This row is wrong and must be closed unbuilt if, at the time it is picked up, a fresh run of the three reproductions above returns a non-empty digest_exceptions() for any of: an option-sleeve payload carrying a failed order, corrupt_position: True, or all-routines-missed: None. One command, one run, no judgement. The reproductions are in this file verbatim so the check cannot drift into a re-argument.

Conversely it is built and done when those three payloads each return a non-empty exception list, the option section renders its failed orders, and the renderer/gate consistency test fails when a new 🔴 line is added to a renderer without a matching exception (show the negative control red).

Effort · horizon

~0.5 person-days (three gather/gate keys plus one consistency test; no trading-path code, observability tier only). Decidable immediately — this is live from today, the first weekday under email_policy: weekly.

Note for triage, disclosed

research/queue/open/ stands at 19 files against its pre-registered 12-file cap before this row. The panel is not pre-empting the overflow rule: rank this against the incumbents like anything else. The argument for it ranking high is only that it is two days old, it sits on the single channel every other alarm ultimately depends on, and it touches the live options sleeve.

Triage note — 2026-09-14 (annotation only; the body above is untouched)

FOS-2 IS FOLDED INTO THIS ROW as of today — same module, adjacent lines, same defect class (the digest's renderer and its send-gate read different sources of truth). FOS-2 was dismissed at the 12-item cap this run; body preserved verbatim at research/queue/dismissed/fos-2-fleet-email-missing-streak-split.md. What it adds here is small and composes exactly: fleet_digest.py:781-785 still renders the flat failed_orders count, while the payload already carries the NEW-vs-STANDING split (digest.py:247-249), so one shared formatter lifted from digest.py:515-528 _failed_detail closes it. Keep FOS-2's negative control in this row's test set: revert the fleet renderer to the flat line, assert red. Estimated ~0.5 pd → ~0.6 pd with the fold.

All three legs re-verified STILL PRESENT at HEAD 340bc5f by reading the full functions: _gather_option_sleeve (:68-172) returns neither failed_orders nor failed_standing (return block :151-172) while digest_exceptions (:1181-1187) reads exactly those two keys; corrupt_position appears nowhere in digest_exceptions (:1157-1189) nor in any_red (:495-505) though the renderer prints it (:805-806); and _gather_routine_liveness:311-325 sets "missed": None, which digest_exceptions:1174-1178 filters as falsy. TECH_DEBT.md:687-688 carries the second at LOW, so the body's "#163 silently promoted a LOW item" is confirmed.

All three legs are LATENT, measured — and that is the argument FOR building now, not against. data/state/vrp/order_log.jsonl holds 52 records with statuses {None: 41, PENDING_NEW: 11} and zero failed across the sleeve's entire life since 2026-07-13, so leg 1 has never been exercised; leg 2 is backstopped by vrp_daily._read_position raising on corruption; leg 3 is backstopped today by fleet-digest.yml's GH_TOKEN. Nothing has fired, which is precisely why this is cheap this week.

The folded FOS-2 half, by contrast, is ARMED — 15 consecutive AVB failed exits through 2026-09-11 render on the next email including meanrev.

Step 3 (a renderer↔gate consistency test) is what makes this defect class non-recurring and should not be dropped in favour of three key additions.