Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

DISMISSED — RCX-1 — portfolio reconcile reports local corruption as "broker unreachable" and exits 0, and the comment saying that was fixed describes a mechanism that cannot work

status: dismissed at adversarial review, panel run 25 (2026-09-17) · class considered: inert guard / loudness (G2) · surfaced by the change-review lens while verifying PR #171's own cost argument, so it is outside the delta — the lines are unchanged since long before this run · the defect is real and verified; it is dismissed on reachability and slot cost, with the comment fix handed over


What was considered, and what is true

thales portfolio reconcile — the designated book-integrity detector, run if: always() after the trade step and the place RUNBOOK's "when a run goes red" sends the operator — swallows local state corruption, mislabels it as a broker problem, and exits 0 (a green step).

Verified at HEAD dfde575:

  • src/thales/cli.py:4205 opens try:;
  • :4215-4218 carries the comment "Read local state FIRST so a corrupt/torn local JSONL raises OUT of the broker-only except below (adversarial review 2026-07-18: the blanket except mislabeled local corruption as 'broker unreachable' and exited 0 — a GREEN reconcile on a corrupt book)";
  • :4219-4220 are the two local reads — inside that same try;
  • :4222 is except Exception as e:, unconditional; :4226-4227 print "reconcile skipped — broker unreachable" and raise typer.Exit(code=0);
  • src/thales/execution/state.py:64-74 — _read_jsonl calls bare json.loads, so a torn line raises JSONDecodeError ⊂ ValueError ⊂ Exception.

Moving a statement earlier within the same try cannot change which handler catches it, so the stated remedy is a no-op against its stated failure mode. The correct shape exists 50 lines away in the same file: backfill-equity (:4272-4280) wraps only the broker call, with the local read outside it. No test pins any of this (grep -rn "broker unreachable" tests/ returns only test_go_live.py's safety-halt strings).

Two refinements found during review:

  • The message is wrong in both directions. _make_broker(...) is outside the try, so a genuine credential/construction failure is never caught by this handler at all; it only ever covers failures inside reconcile().
  • The 2026-07-18 attribution is NOT-VERIFIED. This checkout is shallow (git log -L 4215,4220 bottoms out at 2026-09-09), so what that review actually changed cannot be read here. The "a fix is on the record as having closed a hole it did not close" argument rests on the comment's own text, not on the reviewed diff — and is recorded that way deliberately.

Why it was dismissed

Reachability is engineered away at both writers, and that was measured, not assumed. state.py:47-61 _append_jsonl flushes + os.fsyncs and argues a single JSON line under PIPE_BUF (~4 KiB) is written atomically by POSIX; atomic_write_lines (:31-45) uses temp-file + os.replace. Measured on the real committed logs: data/processed/order_log.jsonl 544 lines / 650 B max, data/processed/equity_history.jsonl 123 / 86 B, data/state/meanrev/order_log.jsonl 3,984 / 403 B, data/state/vrp/order_log.jsonl 55 / 325 B — every record ~6× under the limit.

That leaves one live path: a hand-edit of a committed state file, which is not hypothetical (PR #125, 2026-08-31, corrected the 08-28 phantom equity mark by deleting a line by hand and re-running backfill-equity). But in that scenario the trading run reads the same state early and reds loudly, so reconcile's mislabel costs diagnosis time, not detection.

A one-line typed except is not worth a slot against a 12/12 cap. The dismissal is free and keeps the tripwire.

Handed over instead

Fix the comment even if the except is never touched. A comment that credits a review with closing a hole it did not close is the part that spends trust, and it costs nothing to correct. Suggested for TECH_DEBT.md so it gets done in passing rather than waiting on a queue slot: narrow the try to the broker call (mirroring backfill-equity 50 lines below), or type the except — and either way delete or correct the claim at cli.py:4215-4218.

Reopens if

  • (a) any committed state JSONL line exceeds 4096 B — one awk over data/**/*.jsonl, cheap enough for triage or capture-QA to run — because that is the moment the writer-side atomicity argument stops holding and torn lines become reachable; or
  • (b) a local read failure is ever observed reported as "broker unreachable" in a run log — the true confirmation; or
  • (c) this over-broad-except-around-a-broker-call shape is found at a second site. The shop's own record says this class recurs (IGD-1, AMG-4, PR #168's dark-routine branch, DGX-1 leg 3), and a second instance is what would justify a structural control — a test asserting that every except Exception wrapping a broker call names its exception types — rather than a one-liner.

Deliberately NOT a reopen condition: "the owner hand-edits a state file again." No instrument can observe it, and it asks him to remember a queue row at the exact moment he is doing something risky. The comment fix above is the mitigation for that path.

Prior art checked

No rcx-* slug anywhere in the queue tree or the frozen research/RESEARCH_QUEUE.md. Not covered by IGD-1 (built — five discarded bool returns, a different shape), OSR-1 (open — broker terminal-status blindness), EXQ-1 (built — batch-quote fail-open), or PIT-1 (open — the nearest class, not this instance).


Panel run 45 append (2026-10-27) — reopen condition (c) has FIRED, over-satisfied ten times over, and the second site makes yesterday's brand-new not_found / lookup_failed split unreachable

Recorded, not reopened: dismissed/ → open/ is the steward's move (ops/STEWARD.md:269-270 — "Reopen a dismissed item only when its own written reopen condition has fired — cite it"). This append is the citation that move needs. Precedent for a dated append rather than a mint: rdr-1's own 09-25 annotation on its fired leg (b), plus b4-notch, bcn-1 and lck-1.

The condition, verbatim from :88-94 above: "(c) this over-broad-except-around-a-broker-call shape is found at a second site. The shop's own record says this class recurs (IGD-1, AMG-4, PR #168's dark-routine branch, DGX-1 leg 3), and a second instance is what would justify a structural control — a test asserting that every except Exception wrapping a broker call names its exception types — rather than a one-liner."

The second site, verified at HEAD 781343e:

src/thales/execution/alpaca_broker.py:428-432
    def get_order_by_client_id(self, client_order_id: str) -> Order | None:
        try:
            result = self.client.get_order_by_client_id(client_order_id)
        except Exception:
            return None

It is not one second site. grep -c 'except Exception' src/thales/execution/alpaca_broker.py → 10. Condition (c) asked for a second instance to justify a structural control; the file carries ten, so the control is the proportionate remedy and the one-liner is not.

Found by the change-review lens while reviewing #285, which touched this exact function (for the multi-leg symbol=None crash) and correctly left the handler alone — the TypeError it fixed was raised below the try.

The consequence: a transient broker failure is reported as an unresolvable book-vs-broker divergence, with order ids named

  • status_sweep.py:180-186 files a None return as rep.not_found under the comment "NOT a lag. We recorded this order as submitted and the broker cannot find it under the id we minted, so one of the two records is wrong and no future sweep will resolve it either."
  • cli.py:4366-4374 (thales portfolio sweep-status) prints that count with "these cannot be resolved by a later sweep" and names up to 20 client_order_ids; cli.py:1119-1122 (thales run) prints the same summary and names up to 10.
  • thales run sweeps after trading every weekday on tbx-win, three sleeves.

So a 429 after alpaca-py's own retries exhaust, a read timeout from the deliberately injected 30 s deadline at alpaca_broker.py:54-67, a 5xx, or an expired key reaches the operator as a permanent divergence. The truth is the opposite: the next sweep would have resolved it. And the real divergence class — what OSR-1 was built for — is diluted by transients in the same bucket.

Measured through the real AlpacaBroker, twice independently (change-review lens, then the adversarial reviewer):

injected at the SDK clientsweep reports
APIError 429 rate limitnot found, not lookup error + the id named
read timeoutnot found
APIError 401 × 33 not found, 0 lookup error(s) + three ids named
a broker object that raisesthe only way lookup_failed fires
APIError("order not found") vs APIError("500 internal server error")byte-identical reports

Grounding suites green on the unmodified tree: tests/test_execution/{test_alpaca_broker,test_failed_streaks,test_status_sweep}.py → 75 passed; {test_digest,test_fleet_digest,test_pipeline,test_vrp_fidelity,test_state}.py → 115 passed.

lookup_failed is NEWLY unreachable — it worked for one day, by accident

Stated precisely, because "structurally dead" would be wrong. #285's PR body: "8/8 VRP orders since 09-04 resolve to FILLED. Before the fix: 0, with 8 errors." Pre-#285 the TypeError was raised at the result.symbol use, outside the try that wraps only the client call, so it propagated to the sweep's own handler — lookup_failed was the one bucket VRP ever reached. #285 fixed that bug and in doing so removed the sole production path into the bucket: AlpacaBroker now never raises, and SimulatedBroker.get_order_by_client_id (simulated.py:107-108) is a dict.get. The split did not fail to work. It worked for a day and then sealed itself, the day after the reporting it feeds was merged (#270, 2026-10-26).

Why this matters beyond the mislabel: a memo pinned a claim on this instrument

research/2026-10-22_a_sweep_that_resolves_nothing_says_nothing.md names two indistinguishable causes for the VRP silence — "Alpaca does not resolve a multi-leg parent … so the lookup returned None" vs "the lookup raised and was swallowed" — and says the fix is "deliberately the one that makes the next run answer the question by itself." It cannot: both causes land in not_found. The question was in fact answered by #285's credentialed dry run, not by the instrument built to answer it.

And the gate that waits on that answer is still worded as if the instrument will supply it: fleet_digest.py:1553-1557, inside digest_exceptions (def at :1465 — not _gather_option_sleeve, which is DGX-1 leg 1's subject and a different defect), reads "…which is what will answer the lookup question. Wire this line only after it is answered." The precondition cleared on 2026-10-26 by a credentialed run nine seconds before that comment merged. Recorded here rather than on DGX-1 because the fact belongs to the instrument, not to the option gather's missing key.

Discharge criterion — rewritten at adversarial review, because (c)'s own wording has a wriggle

"Names its exception types" could be satisfied by narrowing :432 alone while nine except Exception remain, or by naming a type in a comment. So:

Discharged only when a test enumerates every except clause in src/thales/execution/alpaca_broker.py whose body wraps or follows a broker call and asserts each names concrete exception types (not bare Exception), and thales portfolio sweep-status reports a transient broker failure in lookup_failed and never in not_found, proven by a test that injects a 429 through the real AlpacaBroker and asserts the bucket. Not discharged by fixing :432 alone: the count at filing is 10 except Exception in that one file, and the test must fail if a new one is added.

The "no future sweep will resolve it either" comment at status_sweep.py:182-184 must be deleted or qualified in the same change, or the code keeps asserting something false.

Remedy and cost, unchanged from what this file already specified: type the handler (or return a distinct sentinel / re-raise a typed error) so a transient lands in lookup_failed and a genuine absence in not_found — ~0.15 pd — plus (c)'s own structural control, ~0.1 pd. Observability tier; no trading-path behaviour change; sweep_terminal_statuses keeps its signature and return.