Thales
← research journal

An internal research document, published verbatim by the automated daily export — not written for an audience, and better for it. All performance discussed is simulated paper trading; nothing here is investment advice.

OSR-1 — 130 market replacements were REJECTED by the broker after the local log recorded them ACCEPTED, and no channel can see a broker-side terminal rejection: the order log is write-once at submit, the digest counts local failed only, reconcile compares ID sets

status: open · raised 2026-09-05 (maintainer verification of the external design review r2 — a side finding from pulling broker truth for its finding 1; NOT in the review) · class: truth divergence between layers (the AI-native audit's recurring class) · judgement: YES · effort ~0.25–0.5 pd · horizon: fired on the 08-28 and 08-31 displaced-cron sessions; recurs on any post-close submission, and applies to every sleeve's exits


Plain-language summary for an owner reading one paragraph. The system writes down an order's status once, at the moment it hands the order to the broker. If the broker later refuses it, nothing comes back to correct that line. On the two displaced-cron days, 130 of the market orders that replaced unfilled limits were refused by Alpaca at the next morning's pre-market processing. The local log says ACCEPTED for every one of them; the daily digest counted zero failures (it only counts orders that failed at submission); the reconciler reported no drift (it checks that the two sides hold the same order IDs, not what became of them); and the 08-31 displaced-cron memo assumed those orders would "fill at the next open". The positions self-heal because each run recomputes from the broker's actual positions, so the economic effect on the never-promotable meanrev control was a one-day delay. But the blindness is structural and sleeve-agnostic: a broker-refused exit on momentum would look identical — green digest, clean reconcile, ACCEPTED in the log.

Ground truth (read-only broker pull, 2026-09-05; 45-day window 07-23 → 09-04, 3,255 broker orders)

  • local meanrev replacements (market_replace_of set): 546, of which 544 in window;
  • replacement terminal statuses at the broker: filled 412 · rejected 130 · expired 2 (2 predate the window);
  • rejected, by local batch date: 08-28 → 45 (broker submitted_at 2026-08-31T11:00Z, i.e. Monday 07:00 ET pre-market) · 08-31 → 85 (2026-09-01T11:00Z);
  • side: 90 buy / 40 sell · type market · all fractional quantities;
  • local order_log.jsonl status for all 130: OrderStatus.ACCEPTED;
  • the fresh selection limit orders of the same two batches: none rejected — only the post-close market replacements were;
  • cause: the orders endpoint carries no reject reason; not determined here — see data plan. Not recorded anywhere in the repo before this row (the 08-31 memo counts the batches and assumes fills).

Why nothing sees it — verified at HEAD cd571b2

  • state.log_order is append-only; the replacement's record is written at submit with "status": res.status (daily.py:1252-1258), the batch's the same way in pipeline.py. There is no status-update record type.
  • digest.py:114: failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"] — local submit-time failures only; the !! N FAILED orders line (:550-551) that the 07-21 audit added cannot count these.
  • reconcile.py:135-160: symmetric difference of order-ID sets; status is carried in the report rows and never compared.
  • FOS-1 (open) computes per-symbol streaks from the same local log, so it inherits the blindness.
  • ops/DAILY_AUDIT.md:146 asks the daily audit to check failed/rejected orders against broker truth (portfolio orders) — the routine was quota-starved on exactly those days (self-audit 2026-09-02, M1), which is the only reason this was found by hand.

Fix shape (propose-only)

  1. Terminal-status sweep, at the start of the next run or in the digest gather: for local orders from the prior N sessions whose recorded status is non-terminal, fetch broker status by client_order_id (AlpacaBroker.get_order_by_client_id already exists) and append a status_update record — the log stays append-only, and the sweep is idempotent per (order, terminal status).
  2. Digest: count broker-terminal rejected / expired (and canceled not initiated by us) as failed; FOS-1's streak logic reads the updated status, which closes its inherited blind spot for free.
  3. Reconcile: a status-divergence section — local non-terminal vs broker terminal — report-only, so the RUNBOOK's step 2 actually answers "did what we sent happen?".
  4. Data plan for the cause: read the rejected orders' failed_at / account activities, or reproduce on the paper account by submitting one fractional market order after the close. Record the reason on this row before designing any behavioural change (e.g. holding replacements until the session opens is a policy change that goes through the owner).

Test design + negative control (CQA-1 doctrine)

  • SimulatedBroker fixture where one submitted order transitions to rejected after submit → the next run's sweep appends a status_update and the digest renders 1 FAILED; revert the sweep → 0 FAILED (today's behaviour).
  • Reconcile fixture with the same divergence → the new section names the order; revert → "OK — local matches broker".
  • Idempotency: running the sweep twice appends one update, not two.

Kill criterion — pre-registered

If the sweep shows the broker exposes a reject reason that explains all 130 as one benign class already healed by re-selection, and the owner rules the one-day delay acceptable for the control sleeve, the policy question closes — but legs 1–3 still stand: the defect is that no layer could see it, not that it happened.

Cap note for triage

Fold partner FOS-1 (same digest surface, same log; leg 2 is literally its missing input). The reconcile leg must survive any fold.


Triage note — 2026-09-07 (annotation only; the body above is untouched)

FOS-1 shipped, and it shipped ON TOP of this row's blind spot. Leg 2 is no longer merely a fix — it is now load-bearing for an alarm that is live.

This row's own cap note said "Fold partner FOS-1 (same digest surface, same log; leg 2 is literally its missing input)." FOS-1 was built instead, alone, in PR #154 (ee68ebe, 2026-09-06) and now sits in queue/built/ recorded as shipped. Verified leg by leg at HEAD e25f80b#154 closed none of this row's four legs:

  • Leg 1 (terminal-status sweep) — not closed. #154 modified cli.py, digest.py, failed_streaks.py (new), pipeline.py and tests; state.py, daily.py and reconcile.py were not touched. There is still no status_update record type in state.py, and nothing calls get_order_by_client_id to refresh a recorded status. The market replacement is still written once at submit with "status": res.status (daily.py:1385-1392).
  • Leg 2 (digest counts broker-terminal rejections) — not closed, and now worse. digest.py:114 is byte-for-byte the line this row quotes: failed = [o for o in todays_orders if str(o.get("status","")).lower() == "failed"]. Everything #154 added reads from that same list — the streak machinery runs only if failed: (digest.py:121) and failed_streaks._index classifies on the same local field (failed_streaks.py:66). A broker-side rejection leaves OrderStatus.ACCEPTED on the record, so it enters neither failed, nor failed_new / failed_standing, nor the new exit-4 escalation (cli.py:1138-1163).
  • Leg 3 (reconcile status divergence) — not closed. reconcile.py:133 still merely carries "status" into the report row; the comparison is still the symmetric difference of order-ID sets (:137,:140,:144,:165).
  • Leg 4 (the cause of the 130) — not closed. pipeline.py:355-356 now keeps the broker's rejection text, but only on the submit-time record["status"] = "failed" branch (:354) — the AVB case. Nothing records why an order accepted at submit was refused later.

Why this raises the row rather than lowering it. Before #154 the blindness was a reporting gap. Now the system's newest alarm — the one that reds the run (exit 4) when a de-risking order fails — is built on the field that cannot see the failure class this row measured 130 instances of. The row predicted exactly this ("FOS-1 ... inherits the blindness"); it is no longer a prediction. And 2026-09-08 is the first live session to exercise the exit-4 path at all, so its first exercise is a blind one.

Ground truth unchanged and not re-pulled this run (no broker credentials in this container): the 130 rejections of the 08-28 and 08-31 batches are recorded in the row above from a credentialed pull on 2026-09-05.