Thales
← research journal
raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

FIN-3 — the OOS monitor's verdict vocabulary cannot say "computable but uninformative", so it printed OK the day the sample crossed 60 while live Sharpe was -1.33

status: open · raised 2026-09-10 (external financial review F4, confirmed — see research/2026-09-09_financial_review_verification.md) · class: loudness & calibration · judgement: YES · effort ~0.25 pd · horizon: must land before the December gate

Plain-language summary. On 09-01 the monitor said INSUFFICIENT_DATA at 56 live returns. On 09-09 it said OK at 61. Nothing about the evidence improved — live Sharpe moved from -1.70 to -1.33, both deeply negative. The sample crossed an integer threshold and the label changed from an honest statement of ignorance to a word that reads as reassurance. The monitor's own design note says "do not false-reassure."

Mechanism

MIN_OOS_DAYS = 60, with the comment "below this the Sharpe CI is so wide the test has no power; declaring OK or DEGRADED would be theatre." The threshold is right and the mechanism is defensible — OK means the live metric has not fallen below the in-sample bootstrap CI lower bound, and at n=61 that bound is +2.830, so almost nothing could breach it. But the project's own interpretation memo (research/2026-07-13_december_gate_interpretation_memo.md) puts the power at ~12% even at 126 days. At 61 it is far lower. The vocabulary (OK / DEGRADED / INSUFFICIENT_DATA) has no way to say "we can now compute a number and that number is uninformative."

Qualification the review omitted, recorded so severity is not overstated: no machine reads this OK as a pass. thales go-live-gate evaluates a separate criterion, forward_sharpe_lower, which FAILS (95% lower -5.456 < 0). The false-reassurance risk is to a human skimming a verdict word.

Fix shape (propose-only)

Either a fourth state (UNDERPOWERED / COMPUTABLE_LOW_POWER) that reports the number with its power beside it, or raise MIN_OOS_DAYS to where the interpretation memo says the test has power. The first is preferable: it keeps the number visible instead of withholding it.

The timing constraint is the point

Do this before December. A verdict-vocabulary change made while looking at the gate's result is gate-shopping, however well intentioned. That is the whole argument for doing it now rather than when it matters.

Test design + negative control

A fixture at n just above the minimum with a wide CI must produce the new state, not OK. Negative control: restore the three-state vocabulary and assert the fixture goes back to OK.

Kill criterion (pre-registered)

If adding a state would require touching the pinned DEGRADED criterion that the retire-rule counts, the item drops to the MIN_OOS_DAYS change instead — the retire criterion's wording is pinned and this row may not soften it.

Triage note — 2026-09-14 (annotation only; the body above is untouched)

One claim in the body above is FALSE, and correcting it raises this row's severity rather than lowering it. The body says "no machine reads that OK as a pass" (the same sentence is repeated in RESEARCH.md's 2026-09-09 banner). Verified at HEAD 340bc5f, two machines read it:

  1. The December gate scores it as a pass. execution/go_live.py:259-277 is criterion 4, oos_monitor_verdict, and its whole test is verdict_ok = verdict != "DEGRADED" (:265). The 2026-09-09 OK therefore marks criterion 4 ok=True on the pre-registered go-live gate. That the gate fails elsewhere (forward_sharpe_lower at 95% lower −5.456) does not un-pass criterion 4 — it means one failing criterion is currently carrying the whole gate.
  2. It resets the incumbent retire-criterion. go_live.py:134 consecutive_degraded() counts the trailing DEGRADED streak and explicitly skips INSUFFICIENT_DATA so a data gap cannot break it — but an OK does break it, and criterion 7 (:330-336) is the two-strikes rule that retires the strategy. So the vocabulary defect does not merely mislead a reader; it consumes a strike the shop pre-registered as its kill switch.

The ledger now permanently reads {"date": "2026-09-09", "verdict": "OK", "n_live": 61, "live_sharpe": -1.3308} (results/diagnostics/oos_verdict_history.jsonl, the only non-INSUFFICIENT row in its history).

The timing constraint in the body is the binding one and this note does not soften it: the change must land BEFORE 2026-12-01. A verdict-vocabulary change made while looking at the gate's own result is gate-shopping.

For whoever builds it: the go_live.py:265 coupling is additive-safe — a new fourth state reads as not-DEGRADED, i.e. status quo — but that is exactly the coupling that must be checked and reported rather than assumed.

Triage annotation — 2026-09-28 (annotation only; the body above is untouched)

Still live at HEAD 864da4e. oos_monitor.py:53 MIN_OOS_DAYS = 60; the vocabulary is still three-valued (:67-69). Both machine readers the 09-14 annotation identified are intact: execution/go_live.py:265 verdict_ok = verdict != "DEGRADED" (criterion 4, :259-277) and go_live.py:134 consecutive_degraded() feeding criterion 7 (:330-341) — so an OK earned at n=61 still both scores criterion 4 as passed and resets the pre-registered retire counter.

The verdict ledger is frozen, and that is the week's only change. results/diagnostics/oos_verdict_history.jsonl still holds exactly 4 rows, unchanged since 09-09, with {"date":"2026-09-09","verdict":"OK","n_live":61, "live_sharpe":-1.3308} the only non-INSUFFICIENT row. The live sample has grown to 68 returns (69 equity marks, 2026-06-11 → 2026-09-18) and is frozen there by the billing outage. 126 by 2026-12-01 remains unreachable — independently re-derived by panel run 29, which counts 120 NYSE sessions from 2026-06-11 to 2026-12-01 inclusive, so n_live maxes at ~119 against min_live_trading_days: 126. not_before is a floor, not a target, and config/settings.yaml:353 already says the 126th lands ~2026-12-10.

The nearer deadline is the next monthly write, not December. Once Actions resumes (allowance resets 2026-10-01) the ~10-01 oos-monitor run appends a second verdict against a live Sharpe that was −1.33 at the last read. If it scores OK on the same underpowered basis, that is a second reset of the retire counter and a second criterion-4 pass — i.e. each restored month makes the December read weaker evidence, not stronger. Fixing the vocabulary after seeing a December result is gate-shopping; that is why this row has a hard 2026-12-01 deadline (64 days) and why its cost is ~0.25 pd.

Blocked on: nothing. Pure build — one enum value, one report line, and a go_live coupling test. The kill criterion still applies: go_live.py:265's != "DEGRADED" is additive-safe against a fourth state, but that must be asserted in a test rather than assumed, or the fix silently changes what criterion 4 means.

Triage annotation — 2026-10-05 (annotation only; the body above is untouched)

THE DEFECT HAS NOW FIRED A SECOND TIME, which is a severity increase, not a repeat. The verdict ledger results/diagnostics/oos_verdict_history.jsonl reads, in full:

2026-07-29 INSUFFICIENT_DATA n=32 sharpe -3.971
2026-08-01 INSUFFICIENT_DATA n=34 sharpe -2.344
2026-09-01 INSUFFICIENT_DATA n=56 sharpe -1.699
2026-09-09 OK                n=61 sharpe -1.3308
2026-10-01 OK                n=75 sharpe -1.3209   <-- NEW

Last week's annotation said this was "the only non-INSUFFICIENT row in its history". That is now false — there are two. The ledger records two consecutive monthly OK verdicts on a live Sharpe that did not improve (−1.3308 → −1.3209: flat, and deeply negative). The monitor has now printed the word that reads as reassurance twice while the evidence stood still. And because src/thales/execution/go_live.py:134 consecutive_degraded() skips only INSUFFICIENT_DATA, each OK resets the retire streak that criterion 7 (:330-336) counts — so a first DEGRADED now needs two more months on top of a freshly-reset counter before the incumbent kill-criterion can fire.

Vocabulary unchanged at HEAD 502690b: exactly three states at src/thales/backtest/oos_monitor.py:67-69, branching at :435 / :450 / :462, MIN_OOS_DAYS = 60 at :53. No UNDERPOWERED or equivalent exists anywhere. The coupling stands at go_live.py:265 verdict_ok = verdict != "DEGRADED". There IS an honest low-power paragraph (oos_monitor.py:304 power_note, rendered :481, :507-525, "A passing OK does not certify the edge") — but it is a separate report field, not a verdict state, and the vocabulary is what both machines read.

Distance to the deadline: 57 calendar days (2026-10-05 → 2026-12-01), i.e. at the monthly cadence about two more verdict rows, both of which will be computed at n < 126 under the same three-state vocabulary. Live returns available today: 79 (80 rows in data/processed/equity_history.jsonl from the pinned clean start 2026-06-11, config/settings.yaml:412-417), against the 126 the pre-registered interpretation memo prices at ~12% power.

One behaviour change a builder must state explicitly rather than discover: a fourth state reads as not-DEGRADED at go_live.py:265, so criterion 4 is unaffected — but it would also not reset the retire streak at :134. That is the real decision inside this 0.25 pd, and it should be written down before the build, not after.

Rank this week: 5 of 12 (P1, and the highest-ranked row whose deadline is the December gate itself). BLOCKED on: owner approval only.

Triage annotation — 2026-10-19 (annotation only; the body above is untouched)

No new verdict since 2026-10-01, and the live evidence has got slightly worse. results/diagnostics/oos_verdict_history.jsonl still holds five rows — 07-29 INSUFFICIENT n=32 −3.971 · 08-01 INSUFFICIENT n=34 −2.344 · 09-01 INSUFFICIENT n=56 −1.699 · 09-09 OK n=61 −1.3308 · 10-01 OK n=75 −1.3209. Two consecutive OKs, and each one resets the retire streak that go_live.py:134 consecutive_degraded() counts (its docstring at :136-137 still skips only INSUFFICIENT_DATA). Next monthly write ≈ 2026-11-01.

Re-derived from data/processed/equity_history.jsonl on the clean clock (config/settings.yaml:423, since: 2026-06-11): 90 equity marks, 89 returns, annualized Sharpe −1.344, equity 14,195.25 → 13,653.21 (−3.82% cumulative). So the 10-01 reading of −1.3209 has moved down to −1.344, and the 10-05 annotation's "79 live returns" is now 89.

The defect is unchanged. oos_monitor.py:53 MIN_OOS_DAYS = 60; exactly three verdict states at :67-69, assigned at :435 / :450 / :462; grep -n "UNDERPOWERED\|COMPUTABLE" over the file → zero hits. The honest low-power text is still a report field, not a state (power_note declared :304, computed :481, emitted :507), so no machine reader can act on it: go_live.py:265 is still verdict_ok = verdict != "DEGRADED".

The December arithmetic, since this row's horizon is "before the gate". 2026-12-01 is 43 calendar days / 30 NYSE sessions away. At 90 marks plus 30 sessions the gate opens with ~120 live trading days against go_live.min_live_trading_days: 126 — so that criterion is unreachable on 2026-12-01 by construction; the 126th mark lands ~2026-12-09, which is what config/settings.yaml:353 already says. That does not change this row, but it does mean the gate will be read with an OK verdict standing on a Sharpe of −1.34, which is precisely the false-reassurance this row exists to prevent.

Live state: blocked on approval only — no data, no credentials, pure build (~0.25 pd). Note the gate-holder changed: under the 2026-10-07 delegation this is the steward's approval, and its 10-16 list names CQA-3 then PLB-1 ahead of this row. Deadline not fired; 43 days.