Thales
← research journal
raw markdown ↗

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

OSR-1 — 130 market replacements were REJECTED by the broker after the local log recorded them ACCEPTED, and no channel can see a broker-side terminal rejection: the order log is write-once at submit, the digest counts local failed only, reconcile compares ID sets

status: open · raised 2026-09-05 (maintainer verification of the external design review r2 — a side finding from pulling broker truth for its finding 1; NOT in the review) · class: truth divergence between layers (the AI-native audit's recurring class) · judgement: YES · effort ~0.25–0.5 pd · horizon: fired on the 08-28 and 08-31 displaced-cron sessions; recurs on any post-close submission, and applies to every sleeve's exits

Plain-language summary for an owner reading one paragraph. The system writes down an order's status once, at the moment it hands the order to the broker. If the broker later refuses it, nothing comes back to correct that line. On the two displaced-cron days, 130 of the market orders that replaced unfilled limits were refused by Alpaca at the next morning's pre-market processing. The local log says ACCEPTED for every one of them; the daily digest counted zero failures (it only counts orders that failed at submission); the reconciler reported no drift (it checks that the two sides hold the same order IDs, not what became of them); and the 08-31 displaced-cron memo assumed those orders would "fill at the next open". The positions self-heal because each run recomputes from the broker's actual positions, so the economic effect on the never-promotable meanrev control was a one-day delay. But the blindness is structural and sleeve-agnostic: a broker-refused exit on momentum would look identical — green digest, clean reconcile, ACCEPTED in the log.

Ground truth (read-only broker pull, 2026-09-05; 45-day window 07-23 → 09-04, 3,255 broker orders)

  • local meanrev replacements (market_replace_of set): 546, of which 544 in window;
  • replacement terminal statuses at the broker: filled 412 · rejected 130 · expired 2 (2 predate the window);
  • rejected, by local batch date: 08-28 → 45 (broker submitted_at 2026-08-31T11:00Z, i.e. Monday 07:00 ET pre-market) · 08-31 → 85 (2026-09-01T11:00Z);
  • side: 90 buy / 40 sell · type market · all fractional quantities;
  • local order_log.jsonl status for all 130: OrderStatus.ACCEPTED;
  • the fresh selection limit orders of the same two batches: none rejected — only the post-close market replacements were;
  • cause: the orders endpoint carries no reject reason; not determined here — see data plan. Not recorded anywhere in the repo before this row (the 08-31 memo counts the batches and assumes fills).

Why nothing sees it — verified at HEAD cd571b2

  • state.log_order is append-only; the replacement's record is written at submit with "status": res.status (daily.py:1252-1258), the batch's the same way in pipeline.py. There is no status-update record type.
  • digest.py:114: failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"] — local submit-time failures only; the !! N FAILED orders line (:550-551) that the 07-21 audit added cannot count these.
  • reconcile.py:135-160: symmetric difference of order-ID sets; status is carried in the report rows and never compared.
  • FOS-1 (open) computes per-symbol streaks from the same local log, so it inherits the blindness.
  • ops/DAILY_AUDIT.md:146 asks the daily audit to check failed/rejected orders against broker truth (portfolio orders) — the routine was quota-starved on exactly those days (self-audit 2026-09-02, M1), which is the only reason this was found by hand.

Fix shape (propose-only)

  1. Terminal-status sweep, at the start of the next run or in the digest gather: for local orders from the prior N sessions whose recorded status is non-terminal, fetch broker status by client_order_id (AlpacaBroker.get_order_by_client_id already exists) and append a status_update record — the log stays append-only, and the sweep is idempotent per (order, terminal status).
  2. Digest: count broker-terminal rejected / expired (and canceled not initiated by us) as failed; FOS-1's streak logic reads the updated status, which closes its inherited blind spot for free.
  3. Reconcile: a status-divergence section — local non-terminal vs broker terminal — report-only, so the RUNBOOK's step 2 actually answers "did what we sent happen?".
  4. Data plan for the cause: read the rejected orders' failed_at / account activities, or reproduce on the paper account by submitting one fractional market order after the close. Record the reason on this row before designing any behavioural change (e.g. holding replacements until the session opens is a policy change that goes through the owner).

Test design + negative control (CQA-1 doctrine)

  • SimulatedBroker fixture where one submitted order transitions to rejected after submit → the next run's sweep appends a status_update and the digest renders 1 FAILED; revert the sweep → 0 FAILED (today's behaviour).
  • Reconcile fixture with the same divergence → the new section names the order; revert → "OK — local matches broker".
  • Idempotency: running the sweep twice appends one update, not two.

Kill criterion — pre-registered

If the sweep shows the broker exposes a reject reason that explains all 130 as one benign class already healed by re-selection, and the owner rules the one-day delay acceptable for the control sleeve, the policy question closes — but legs 1–3 still stand: the defect is that no layer could see it, not that it happened.

Cap note for triage

Fold partner FOS-1 (same digest surface, same log; leg 2 is literally its missing input). The reconcile leg must survive any fold.

Triage note — 2026-09-07 (annotation only; the body above is untouched)

FOS-1 shipped, and it shipped ON TOP of this row's blind spot. Leg 2 is no longer merely a fix — it is now load-bearing for an alarm that is live.

This row's own cap note said "Fold partner FOS-1 (same digest surface, same log; leg 2 is literally its missing input)." FOS-1 was built instead, alone, in PR #154 (ee68ebe, 2026-09-06) and now sits in queue/built/ recorded as shipped. Verified leg by leg at HEAD e25f80b — #154 closed none of this row's four legs:

  • Leg 1 (terminal-status sweep) — not closed. #154 modified cli.py, digest.py, failed_streaks.py (new), pipeline.py and tests; state.py, daily.py and reconcile.py were not touched. There is still no status_update record type in state.py, and nothing calls get_order_by_client_id to refresh a recorded status. The market replacement is still written once at submit with "status": res.status (daily.py:1385-1392).
  • Leg 2 (digest counts broker-terminal rejections) — not closed, and now worse. digest.py:114 is byte-for-byte the line this row quotes: failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"]. Everything #154 added reads from that same list — the streak machinery runs only if failed: (digest.py:121) and failed_streaks._index classifies on the same local field (failed_streaks.py:66). A broker-side rejection leaves OrderStatus.ACCEPTED on the record, so it enters neither failed, nor failed_new / failed_standing, nor the new exit-4 escalation (cli.py:1138-1163).
  • Leg 3 (reconcile status divergence) — not closed. reconcile.py:133 still merely carries "status" into the report row; the comparison is still the symmetric difference of order-ID sets (:137,:140,:144,:165).
  • Leg 4 (the cause of the 130) — not closed. pipeline.py:355-356 now keeps the broker's rejection text, but only on the submit-time record["status"] = "failed" branch (:354) — the AVB case. Nothing records why an order accepted at submit was refused later.

Why this raises the row rather than lowering it. Before #154 the blindness was a reporting gap. Now the system's newest alarm — the one that reds the run (exit 4) when a de-risking order fails — is built on the field that cannot see the failure class this row measured 130 instances of. The row predicted exactly this ("FOS-1 ... inherits the blindness"); it is no longer a prediction. And 2026-09-08 is the first live session to exercise the exit-4 path at all, so its first exercise is a blind one.

Ground truth unchanged and not re-pulled this run (no broker credentials in this container): the 130 rejections of the 08-28 and 08-31 batches are recorded in the row above from a credentialed pull on 2026-09-05.

Triage note — 2026-09-14 (annotation only; the body above is untouched)

Raised again, for the second consecutive week, and this time by a merge that was not about it. PR #163 (3fc2dba, 2026-09-12) moved the fleet email to weekly-plus-exception-days. The "NEW failed order(s)" exception is derived at fleet_digest.py:1184-1187 from d.get("failed_orders") / failed_standing — i.e. from digest.py:114, the local submit-time field this row exists about.

Consequence, stated plainly: a broker-side rejection can no longer even make a day an exception. Before #163 this class was invisible for one day. It is now invisible for up to a week. No ruling produced that; a cadence change did.

All four legs re-verified OPEN at HEAD 340bc5f, leg by leg:

  • Leg 1 — grep -rn "status_update" src/thales/ returns zero hits; no such record type exists in state.py. get_order_by_client_id exists (broker.py:82, alpaca_broker.py:429) but every caller is submit-time idempotency dedup. daily.py:1385-1390 still writes "status": res.status once and never refreshes it.
  • Leg 2 — digest.py:114 is byte-identical to the body's quote. FOS-1's streak machinery (:115-130) is gated if failed: at :121, so it reads the same local-only field.
  • Leg 3 — reconcile.py:133 carries "status" into the row, but the only comparisons are :143 and :159, a pure ID-set symmetric difference. Status is used for display only (:51).
  • Leg 4 — pipeline.py:354-360 keeps the broker's rejection text only on the submit-time failed branch (the AVB case), not this one.

Fold partner updated. FOS-1 has shipped, so the live fold partner is now DGX-1 — same file, same send-gate, same "renderer and gate read different sources" class. They are distinct: DGX-1 covers corrupt_position and missed=None for the option sleeve; neither covers this. Legs 1–3 are buildable today with SimulatedBroker fixtures and need no owner ruling; only leg 4 (why the 130 were refused) needs credentials.

TRIAGE ANNOTATION 2026-09-21 — unchanged and still the cheapest P0 in the queue

All four legs re-verified at HEAD a325c3c: grep -rn "status_update" src/thales/ returns zero hits (leg 1); digest.py:114 byte-identical failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"] (leg 2); reconcile.py:142-144 and :159-165 are still pure order-ID set differences (leg 3); pipeline.py:354-359 still records the broker's rejection text only on the submit-time failed branch (leg 4). No merged commit since 09-14 touched any of them.

This is one of the three rows that need no ruling from the owner at all — only the filesystem move. Legs 1-3 build against execution/simulated.py with no credentials; only leg 4's forensics needs a broker pull. It has been named as build-ready in three consecutive triage reports.

Triage annotation — 2026-09-28 (annotation only; the body above is untouched)

Re-verified at HEAD 864da4e: all four legs open, byte-identical to the 09-21 reading. Still the most build-ready row in the queue. Leg 1: grep -rn "status_update" src/thales/ returns 0 hits; daily.py:1388 still writes "status": res.status once, at submit. Leg 2: digest.py:114 unchanged — failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"], with the FOS-1 streak block gated on if failed: at :121. Leg 3: reconcile.py:133 carries "status" for display only; the comparisons at :142-144 and :157-165 are pure ID-set differences. Leg 4: pipeline.py:354-360 attaches record["error"] only on the submit-time "failed" branch.

Compounding with FOS-3, and the fix sites do not overlap. This row shows the exit-4 alarm reads a field that cannot see broker-side rejections; FOS-3 (annotated this run with live measurements) shows the same alarm has a second, independent silence — a side-blind mask at failed_streaks.py:73. Two blind spots, one alarm, disjoint fixes (digest.py:114 + state.py here; failed_streaks.py there). Whoever builds either should read both.

The 130-rejection ground truth could not be re-pulled (no broker credentials in this container; it dates from a 2026-09-05 credentialed pull) and nothing in committed state contradicts it — data/state/meanrev/order_log.jsonl still holds OrderStatus.*-prefixed statuses, consistent with the write-once claim.

Blocked on: nothing for legs 1-3 — buildable against execution/simulated.py with no credentials. Leg 4 (why those 130 were refused) needs a broker pull, i.e. the owner's environment, and should be split rather than allowed to block the rest. Honest effort: legs 2+3 ~0.2 pd; leg 1's idempotent sweep plus a status_update record type and three fixtures ~0.4 pd; ~0.5 pd total — the body's 0.25 low end is optimistic. Fold partner remains DGX-1 (same file, same send gate).

Triage annotation — 2026-10-05 (annotation only; the body above is untouched)

All four legs open at HEAD 502690b, and the premise was exercised again TODAY. Leg 1: grep -rn "status_update" src/thales/ returns zero; src/thales/execution/daily.py:1385-1390 still writes the market replacement once with "status": res.status; every get_order_by_client_id caller is submit-time dedup (alpaca_broker.py:213,251,322, pipeline.py:435); grep -rn terminal src/thales/execution/*.py returns zero. Leg 2: src/thales/execution/digest.py:114 is byte-identical — failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"] — with FOS-1's streak block gated if failed: at :121. Leg 3: src/thales/execution/reconcile.py:133 carries "status" into the row while the comparisons at :142-144 and :157-165 remain pure order-ID set differences (PR #213 changed only the equity-date keying, per its own comment at :167-170). Leg 4: src/thales/execution/pipeline.py:354-359 keeps the broker's text only on the submit-time failed branch. The cadence amplifier is intact at fleet_digest.py:1184-1187, which derives "NEW failed order(s)" from digest.py:114.

Re-measured, and the number in the body is stale in the row's favour: data/state/meanrev/order_log.jsonl now holds 550 market-replacement records (body says 546), with statuses 528 OrderStatus.ACCEPTED + 22 OrderStatus.PENDING_NEW — 100% non-terminal, which is precisely the premise. Replacements landed on 09-11, 09-18 and two on 2026-10-05, so the path is live, not historical. Momentum carries 39, all from 2026-06-11.

Rank this week: 10 of 12 (P2 instrument calibration). It holds its slot on cheapness and breadth rather than urgency: legs 1-3 are blocked on nothing (buildable against src/thales/execution/simulated.py, 0.25-0.5 pd), and the fix applies to every sleeve's exits. Only leg 4 needs a credentialed broker pull, which is the one thing this routine cannot do. The 130 broker rejections remain un-re-verifiable without credentials, as the row itself states. BLOCKED on: owner approval for legs 1-3.

Approved by the steward — 2026-10-08 (legs 1–3 only)

Approved by the steward under the owner's 2026-10-07 delegation: this is P0 under NORTHSTAR §4 (a silent-failure class — the order log records ACCEPTED for orders the broker later rejected, and no channel can see it); it has a testable done-when built on SimulatedBroker fixtures with the row's own negative control; and triage has recorded on four consecutive Mondays that legs 1–3 need no owner ruling, no data and no credentials. Scope is legs 1–3 only (terminal-status sweep as an append-only status_update record; digest counts broker-terminal rejected/expired as failed; reconcile gains a report-only status-divergence section). Leg 4 needs broker access the implementer does not have, and any behavioural change it motivates (e.g. holding replacements until the open) is a policy change that stays with the owner — do not build it. The build touches src/thales/execution/, so its PR will be reviewed under STEWARD.md §3's trading-path rule: it must change no order the sleeves submit, and say so in one sentence.

BUILT — 2026-10-21, implementer, PR #262 — outcome: shipped (legs 1-3)

Built exactly to the steward's 2026-10-08 approval: legs 1-3 only. Leg 4 (why those 130 were refused) is NOT built — it needs a credentialed broker pull this container does not have, and any policy change it motivates stays with the owner, as the approval says.

  • Leg 1 — execution/status_sweep.py + StateManager.log_status_update / read_status_updates. After each run (and via thales portfolio sweep-status), recent orders whose recorded status is still non-terminal are re-asked of the broker by client_order_id, and a status_update record is appended. The order log stays append-only: the submit-time line is never rewritten. Idempotent per (order, terminal status); capped at 300 lookups; every broker call individually wrapped.
  • Leg 2 — digest.py now counts broker-terminal rejected / expired as failed, and canceled only where no record claims the cancel as ours. It scans market-REPLACE records too (the 130 were all replacements, which todays_orders excludes) and reports a failure on the day it was LEARNED, which for a post-close submission is a later date than the order's own. FOS-1's streak machinery reads the corrected history.
  • Leg 3 — reconcile.py gains a report-only STALE LOCAL STATUS section (local non-terminal vs broker terminal). Deliberately not part of has_drift: a stale status is a reporting lag, and redding the trading gate for it would be worse than the disease.

Kill criterion verdict: NOT FIRED, and not evaluable in this scope. The pre-registered criterion turns on what the broker's reject reason shows — leg 4, excluded by the approval. The criterion's own second clause already settles the legs built here: "legs 1-3 still stand: the defect is that no layer could see it, not that it happened."

Trading-path statement (STEWARD.md §3): this changes no order any sleeve submits. The sweep runs AFTER the wrapper has finished submitting and reads only prior sessions' records; nothing in portfolio/construct.py, execution/pipeline.py::execute_orders, execution/safety.py or execution/daily.py is touched.

Still open: leg 4 (forensics, owner's environment). Adjacent defect found while building and FILED, not fixed: VRD-1 (queue/open/) — VRPRunResult has no run_date, so the exit-4 de-risking escalation silently degrades on the option sleeve.