Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

COV-1 — a selection can run on a fraction of the universe and report success

status: built 2026-11-03 at the owner's direct instruction · raised interactive session 2026-11-03 · class: P0 fail-closed · effort ~0.5 pd · horizon: before the 2026-12-01 selection

Plain-language summary. On 2026-11-02 the momentum sleeve "selected" 37 stocks because only 37 had a price for that day. It bought all of them whatever their momentum: only 4 were in the true top 50. The run reported success and nothing went red, so without a person looking it would have held that book for a month. Full account: research/2026-11-03_momentum_selection_voided.md.

Why nothing caught it. The freshness gate checks only the panel's newest bar, and one fresh symbol is enough to pass. compute_ranked_signals then drops every name with no row on the decision date. Three conditions combined: a run after the close, Tiingo's rate limit, and yfinance's exclusive end date. Any run after the close that exhausts Tiingo can repeat it, cold cache or not.

Proposed build.

  1. The guard (the core of this item). On a selection day, before any sizing, refuse to select if the ranked names fall below a pinned share of the universe. Log selection_completed: false so the next run this month retries, and submit nothing. Coverage has never been measured on a good day, so read the threshold from the run history first and pin it before building.
  2. Hold list. Report held names missing from the ranking as a coverage failure. Today compute_diff holds them with one warning each (63 on 11-02).
  3. Same end date from both vendors. Make yfinance and Tiingo return the same last bar for the same request, so a fallback cannot split the panel by a day. This changes which bars the fallback returns, so check it against the parity harness.
  4. Selection after the close. Decide whether a backup leg that starts after the close may select at all, or should leave the month's selection for the next session. This one changes schedule behaviour, so it is the owner's ruling.

Negative control. Replay the 11-02 shape in a test: a panel where 37 of 903 names have a bar on the decision date. The run must submit nothing, record selection_completed: false, and select normally on the next run.

Kill criterion. If normal coverage on real selection days varies too much to separate from a failure at any single threshold, compare against the prior selection's ranked count instead.

Outcome (2026-11-03)

The owner told the session that found this to finish it, so it skipped the queue's approval step.

  1. Guard: built, in a different shape. A selection now refuses if under 90% of recently trading names have a bar on the panel's newest date (execution.min_panel_coverage, keystone-pinned for momentum and meanrev). This measures the panel rather than ranked names, because meanrev only ever ranks its 500 most liquid names. Expected names are the configured universe plus anything with a bar in the last 10 sessions, because a name no vendor returned has no file and a panel alone could not miss it. The check runs before the Kelly booking. A refusal logs selection_completed: false, exits non-zero, and the next session retries. A second gate refuses when fewer names are ranked than the sleeve holds.
    • Threshold evidence: the box's live panel on 10-29 scores 98.1% against the 903-name universe, and the 11-02 run scores 4.1%. Across 511 sessions of that panel, panel-only coverage never fell below 99.5%. That history has been topped up since, so it is not a record of what each past run saw.
    • A cold review of the first version found two defects, both fixed here. A panel of only 55 names scored 100%, which is why the universe is now in the denominator. And a refused kill-switch re-entry lost its selection for the month, because the kill-switch state had already moved on. Refusals on that run now log reentry_pending, which also covers the older stale-panel, no-signals and safety-halt paths.
    • Known limit: a refusal counts as the day's run, so a same-day retry needs --force and GitHub's backup leg will not retry after the box refuses. The next session retries.
    • Known limit: orders from a retried re-entry are tagged selection, not re_entry. This affects the label only.
  2. Hold list: built as reporting. Run summaries now carry n_ranked, panel_coverage and n_held_unranked. The hold rule itself is unchanged, because the gates stop the 11-02 shape first.
  3. Same end date from both vendors: not built. Asking yfinance for one more day would return today's unfinished bar on the box's 10:35 ET run, and that run uses yfinance for most names. That is worse than the problem. The gate catches the split panel instead.
  4. Selection after the close: unchanged. With complete data, orders placed after the close queue for the next open, which matches the backtest's next_open timing. The 11-02 orders were cancelled first, so no such fill has been observed in this incident. Dropping it would make a missed box run worse. A split panel after the close is now refused, and the next session retries.

Not verified: a real refusal. The negative control and the next-session retry are synthetic replays in tests/test_execution/test_panel_coverage.py.


Annotation — daily audit 2026-11-05: leg 4 is not a design question, it is the leg that carries the incident

Leg 4 is written as a schedule preference ("decide whether a backup leg that starts after the close may select at all"). The measurement below says the after-hours case is now the normal case for these legs, not an edge one, which moves leg 4 from "nice to settle" to the leg the 11-02 incident actually turned on.

Queue delay, measured over the last 30 consecutive scheduled paper-trading.yml runs (2026-09-25 → 2026-11-05; delay = created_at − cron, baseline switched at the real cron move, 30 15 → 30 16 on 2026-10-24, commit 13aacfc):

delay
minimum of 303 h 19 m
median~4 h 40 m
maximum7 h 09 m (2026-11-02, this incident)
runs under 2 h0 of 30
last five4 h 28 m · 7 h 09 m · 5 h 24 m · 5 h 35 m · 5 h 42 m

With the post-DST cron at 16:30 Z and the US close at 21:00 Z, a leg reaches the market only if the delay is under 4 h 30 m. Since the cron moved on 10-24, 2 of 9 runs would have been in-market; the other 7 started after the close. So the backup leg is now predominantly an after-hours actor, and the after-hours path is also the one with a cold cache and an exhausted vendor budget — i.e. the three conditions this row names as "combined" are not independent, they are the signature of the delayed leg.

The VRP sleeve shows the same evening from the other side: its 11-02 order was rejected outright with options market orders are only allowed during market hours, so for options the after-hours leg cannot act at all. (That rejection also exposed a separate alarm defect — see DRO-1.)

Implication for the build, offered rather than decided, since leg 4 is the owner's: legs 1–3 make a partial panel safe, but they leave a backup leg that correctly declines to do the one job it exists for. If leg 4 rules that an after-hours leg may not select, the guard in leg 1 becomes the mechanism that enforces it rather than a second check beside it.

Full measurement and provenance: research/2026-11-05_a_liveness_beacon_deleted_its_own_relay.md §2. CLAUDE.md still tells readers the delay ranges "from well under an hour to over two"; the measured floor is above that sentence's ceiling, and the one-line correction is filed in ops/HANDS.md.


ANNOTATION 2026-11-09 (panel run 49) — one sentence in this row, and a docstring it relies on, are false on the kill-switch path: there are TWO Kelly booking sites and the coverage gate precedes only one

Not a reopening and not a new row. The build is sound and this changes nothing about it. Filed here rather than as a row because the reachability is low, the precedent is against a row (dismissed/kbm-1-dateless-snapshot-booking-skip.md and dismissed/ksp-2-corrupt-state-snapshot-intent-false.md both killed sibling Kelly-booking edges as padding), and the real value is that a sentence in a row closed six days ago is wrong — which is exactly what an annotation is for.

The false sentence. This row says "The check runs before the Kelly booking." True on the selection path, false on the kill-switch path. The kill-switch early return calls _book_kelly_returns_before_liquidation(today, prices_df) and then returns, about 68 lines before the coverage gate is reached (src/thales/execution/daily.py — booking at :1592-ish, gate at :1660-1687). The in-code comment "Ahead of the Kelly booking, which reads the same panel" is false in the same way. So on a kill-switch day the half-Kelly holding period is booked from an unchecked panel.

Why that is the specific shape worth recording. _latest_closes (:933-948) resolves "the latest close on or before asof", so a name with no bar today silently books against an older close — which is exactly the corruption the 2026-11-02 repair had to undo by hand: research/2026-11-03_momentum_selection_voided.md names "October's holding-period booking, which mixed 10-30 and 11-02 closes."

And the docstring no longer describes its caller. It says the kill-switch branch "runs on non-selection days, when no panel has been loaded". But src/thales/cli.py:1041-1048 loads the panel for every sleeve whose contract is not order_intent, so prices_df is not None on this path in production. A future reader reasoning from that docstring would conclude the booking cannot see a partial panel, and would be wrong.

Reachability, stated honestly: low. It needs a 20% drawdown day coinciding with a partial panel. The booking is deliberately fail-soft and must stay that way — de-risking is never to be blocked.

If anyone acts on this, the shape is: call panel_coverage before the booking on that path; below the pin, log critical and skip the booking, never the liquidation. ~5 lines. Negative control: a kill-switch run on a split panel asserting the booking is not called and the liquidation is. No test covers a kill-switch day on a partial panel today — tests/test_execution/test_panel_coverage.py stubs _evaluate_kill_switch to False in its unforced harness.

Minimum honest outcome, if nobody builds it: strike the false sentence from this row's body and fix the docstring, so the next reader is not misled.