2026-09-21 — GitHub refused every job in the repo over billing; nothing traded, and nothing said so
Status: INCIDENT, live at the time of writing. Root cause identified and quoted verbatim from GitHub; the fix is the owner's (a billing page), not a code change. Recorded by the daily audit.
What happened
No sleeve traded on Monday 2026-09-21. main carries no commit for the day —
no state, no site export, no capture. Every GitHub Actions run created for this
repository since some point over the weekend has failed, including workflows
that share no code with each other.
The last successful run of any kind was 2026-09-18 22:37:09Z. The first failure was 2026-09-21 13:43:47Z. Nothing ran on 09-19 or 09-20, so the state flipped somewhere inside that weekend window.
Today's inventory, from the Actions API: zero successes. The three trading sleeves were dispatched on time from the owner's Mac at 14:35:08 / :11 / :13Z and all three were refused; their backup crons at 19:40 / 19:42 / 19:44Z were refused; the fleet digest at 20:07Z was refused; the routine beacon relay and the outbox mailer were refused; auto-merge was refused. A push made at 22:18Z while writing this was refused too, so the block was still in force then.
The cause, in GitHub's own words
Every failed job carries the same annotation, which the audit read at
GET /repos/MaksimDan/thales/check-runs/{id}/annotations:
"The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings"
This repository is private, so its Actions minutes are billed against the account's allowance rather than being free. Either the allowance is spent with a spending limit that forbids overage, or a payment on file failed. Both land on the same page, and neither is visible from inside the repo.
The signature, so the class is cheap to recognise next time
This is the third distinct way this platform's jobs have died, and it looks like neither of the first two:
| runner | duration | steps | logs | conclusion | |
|---|---|---|---|---|---|
| ordinary failure | assigned | full | present, one red | present | failure |
| runner loss (2026-08-06) | runner_name: "" | ~15 min | none | none | cancelled |
| platform refusal (today) | runner_id: 0, runner_name: "" | 3–11 s | none | 404 | failure |
get_workflow_run_usage closes it: billable.UBUNTU.total_ms: 0. Nothing ran,
so nothing was billed — and a 404 on logs is not a tooling problem, it is the
absence of a log stream that was never opened.
The discriminator worth remembering is that the annotations endpoint names the cause outright. One API call, no inference. Three separate routines today (panel, weekly audit, daily audit) saw the failures; the queue-triage routine was the one that went and read the annotation, and it is the only place the actual sentence appears.
The part that matters more than the outage: the alarm inverted
The operator's inbox today contained exactly three Thales messages, all
healthchecks.io DOWN notices — thales-panel (17:00Z),
thales-weekly-audit (20:00Z), thales-triage (22:00Z).
All three of those routines ran perfectly. Each pushed its beacon commit on
schedule — beacon/panel 13:47:13Z, beacon/weekly-audit 14:22:15Z,
beacon/triage 16:17:10Z — and each opened its PR. What failed was the
relay: routine-beacon.yml is a GitHub Actions workflow, so it was refused
along with everything else, and the ping never reached healthchecks.io.
Meanwhile the thing that actually broke — three sleeves not trading — produced
no alert at all. The alert / email job inside each trading run is itself
an Actions job in the same refused run (jobs 106377069686 / 106377075520 /
106377093282). The fleet digest that would have reported "SLEEVE DID NOT RUN"
was refused. The heartbeat monitor, the dead-man's switch built for exactly
this, is an Actions job too.
So the alarm layer said, loudly and three times, that the research routines were dead — which was false — and said nothing whatsoever about the trading platform, which was true. A reader who trusted the inbox would have spent the evening on the wrong problem.
This is the structural residual §4b of ops/DAILY_AUDIT.md already named — no
in-band alarm can be present during an outage that takes out the whole band —
except that it now has a sharper edge. The escape hatch §4b nominated was
"healthchecks.io and these cloud routines". Healthchecks fired, and it fired
wrong, because the routine beacons reach it through the very band that was
down. routine-beacon.yml's own header records the two earlier lessons that
forced the relay design (routine containers cannot reach hc-ping.com; they
cannot use repository_dispatch either). Today adds the third: the relay
inherits Actions' availability, so a routine DOWN is only evidence about the
routine when Actions is up.
The Actions-independent arbiter exists and costs one command. Routines push their beacon commits with a contents write, which needs no runner:
git log -1 --date=iso --pretty='%ad | %s' origin/beacon/<routine>
A beacon branch that moved on schedule under a DOWN email means the relay failed, not the routine. That check is now written into the daily audit's procedure (§4c, check 5).
What this cost
- A trading day on all three sleeves. Momentum's daily volatility check and
its drawdown kill-switch did not evaluate — the committed kill-switch state
is still stamped
last_date: 2026-09-18. Had the book breached its −20% threshold today, nothing would have fired. - A decision on the one live sleeve. VRP closed its spread on 09-18 and is flat; the pinned rule re-enters on the next run. Whatever it would have done today, it did not do, inside a 126-trading-day forward clock currently at 38.
- A forward-capture day that cannot be refetched. The skew snapshot never ran. Option skew, chains, IV term structure, borrow flags and the PiT snapshot are point-in-time streams; 09-21 is simply absent from them. The FINRA short-volume leg is backfillable; the rest is not.
- Both report channels. The outbox relay is an Actions workflow, so the triage routine's report and this audit's report were written and not mailed. Both routines fell back to a phone notification.
What was NOT affected
The cloud routines themselves, which run on Anthropic's infrastructure and reach GitHub through the API: panel, weekly audit and triage all completed and opened PRs (#179, #180, #181). The repository is not disabled, the scheduler is alive (it created today's runs), and no workflow file has changed since 2026-09-07. This is an account-level refusal, not a defect in the tree.
Days 2–4 (2026-09-22 → 09-24): the outage held, and the watcher went dark
Appended by the daily audit of 2026-09-24. The paragraphs above were written on day 1 and are left as they stood; this section is what four days changed.
The refusal is unchanged and still account-level. Read again today off the
trade job of the 18:36Z momentum run (check-run 107777545034), the annotation
is the same sentence, word for word:
"The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings"
Every run on 09-22, 09-23 and 09-24 died in 3–11 seconds with zero steps —
schedule-triggered and Mac-dispatched alike. main has carried no commit since
2026-09-18. The Mac dispatcher is healthy and kept firing on time all four days
(workflow_dispatch at 14:35Z, 14:37Z, 14:42Z, 14:52Z); it had nothing to
dispatch onto.
Cost, revised from one day to four. Four trading days on all three sleeves.
Four un-refetchable forward-capture days (thales captures reports all seven
time-gated streams at 4 days stale and not accruing). Four days in which
momentum's drawdown kill-switch did not evaluate. And four days off both
forward clocks — which has a dated consequence: momentum stands at 68 of the
126 live trading days its go-live gate requires, so even if trading resumed on
2026-09-25 the 126th day now falls on 2026-12-16, past the gate's
not_before: 2026-12-01. Each further lost day moves that date one further.
The finding: a dark cloud routine has no detector at all
Day 1's lesson was that the alarm layer pointed away from the failure. Days 2–4 sharpened it into something worse.
The daily audit did not run on 09-22 or on 09-23. Neither did the research
panel on any of the three days, nor the implementer on 09-23. The evidence is
Actions-independent and therefore trustworthy: the beacon branches are written
by git push from the routine container, which needs no runner, and not one of
them moved after 2026-09-21 22:23Z. Two weekday audits are simply missing.
Nothing anywhere reported this, and the reason is structural:
- The fleet digest's dark-routine exception is an Actions job — refused.
- The healthchecks beacons relay through
routine-beacon.yml— also an Actions job, so every check was already DOWN for relay reasons.thales-daily-auditwent DOWN at 2026-09-22 02:00Z and stayed there; healthchecks notifies on the transition, so the genuinely-dark days produced no new mail and could not have. - The cloud routines are each other's only out-of-band witness, and none of them watches the others.
So during an Actions outage the beacon channel loses precisely the distinction it exists to make: "the routine ran and the relay died" and "the routine never ran" present identically. On 09-21 it was the first. On 09-22 and 09-23 it was the second. Nothing in the system could tell them apart.
And §4c, as written on day 1, made that worse. It taught the next reader
that a routine DOWN under a refused-Actions state is a relay artefact — true
that day, and the owner was told so explicitly. Two days later the same signal
was real. An audit that trains its operator to dismiss a class of alarm owes
that operator the symmetric case in the same breath; day 1's text gave the
discriminator (git log origin/beacon/<routine>) but framed only one of its two
outcomes as a finding.
The correction is check 12 below — cheap, self-referential, and the one thing this routine can verify about itself without trusting any layer that an outage can take down.
Residual — narrowed 2026-09-25
No detector inside this repository can catch a dark cloud routine during an Actions outage, because every in-repo detector needs a runner. Check 12 catches it retrospectively, on the next audit that does run — which is a real improvement over nothing, and is not the same as catching it live.
The original wording of this residual — that closing the live gap needs a watcher outside both Actions and the routine scheduler — was too strong, and panel run 28 (2026-09-25) demonstrated it by doing the thing it declared impossible: it queried the routine control plane, which is Actions-independent, and established that the implementer did fire on 09-23 and died after 16 seconds. Beacon evidence cannot distinguish that from never firing. So the fire-vs-die question this audit escalated to the owner on 09-24 is answerable in-band, at least for that routine, and the escalation should not have been framed as requiring him.
What remains genuinely open is narrower and worth stating exactly, because the
two routines disagree about their own reach: the daily audit's session could
not reproduce that query (2026-09-25 — CronList returns only jobs created
by CronCreate within the calling session, not the account's routine roster).
So the capability exists but is not uniformly reachable, and which tool the
panel used is not recorded in its report. Next action, cheap and not the
owner's: have whichever routine can reach the control plane write down the
exact call, so check 12 can escalate from a retrospective branch-date read to a
live liveness query. Only if no routine can reach it does this become the
owner-level architecture decision originally claimed.
Amendment 2026-09-26: which of the two causes it was
The owner read the account's billing page on 2026-09-22 (Settings → Billing & plans, personal account). It resolves the either/or in "The cause, in GitHub's own words" to its first branch: the allowance was spent. No payment failed.
- Plan: GitHub Free, no payment method on file, no payment history, so there was no payment that could fail.
- September (1–30) gross metered usage $12.06, of which included usage covered $12.06: the month's whole free allowance, consumed. By repository: thales $5.37, tesla-toolbox-companion $5.07, docket $1.24, three others $0.38. The allowance is shared by every private repo on the account, so thales did not exhaust it alone.
- With no payment method, overage cannot be billed, so once the included minutes were gone GitHub refused every job. The annotation's wording ("recent account payments have failed or your spending limit needs to be increased") covers both cases; this was the second.
Consequences worth keeping:
- It ends by itself on 2026-10-01, when the next period's allowance starts, hours before that day's 14:35Z dispatch (momentum's monthly selection). No owner click is needed for that.
- The owner has decided to stay on the free tier (2026-09-22). So the
remedy is not "restore billing"; it is "fit inside the free allowance across
all repos". At September's pace thales used roughly 65–70 billable minutes
per trading day, about 1,450 in a 22-session month. The largest single waste
is the backup
scheduleleg of each trading workflow, which re-runs the whole job (about 14 minutes for momentum) after the Mac dispatch leg already did the day's work. Every job is also rounded up to a whole minute, so the many few-second relay and guard jobs cost a minute each. - It will recur whenever the allowance runs out, most likely late in a month. The signature above identifies it from the repo side; the billing page is the only place the numbers are. Routines should stop escalating "check Billing & plans" as the fix; the useful escalation is how close the month is to the allowance.
Amendment 2026-09-28 (daily audit): when it recurs, and why the repo cannot measure its own burn
The 09-26 amendment closes the cause and names the remedy ("fit inside the free allowance") and the right escalation ("how close the month is to the allowance"). Two things it leaves open, both of which matter before the next reset.
1. The queued trims are ~8% of the account's burn, so expect this again around 2026-10-20
The arithmetic, from the 09-26 amendment's own numbers plus the repo's NYSE calendar:
- The allowance covered 13 trading days of September load. Last successful run 2026-09-18 22:37Z; first refusal 2026-09-21 13:43Z; 09-19 and 09-20 were a weekend, so exhaustion fell inside that gap. September had 21 trading days, and 09-01…09-18 is exactly 13 of them.
- thales is $5.37 of the account's $12.06, i.e. 45% of the burn. The other private repos are invisible from inside this one.
- PRs #185 + #186 remove ~250 minutes/month from thales. Against thales' measured pace (~65–70 billable min per trading day, ~1,450/month) that is ~17% of this repo's burn and ~8% of the account's.
- 13 trading days ÷ 0.92 ≈ 14.1. October's 14th trading day is 2026-10-20 (13th is 10-19, 15th is 10-21). October has 22 trading days, so the month's last ~8 trading days would be dark — the same shape as September.
Prediction, recorded so it can be checked: absent a cut materially larger than 8% of the account's burn, Actions is refused again between 2026-10-19 and 2026-10-23, and stays refused to 2026-10-31. Merging #185 and #186 is a real improvement and buys roughly one extra trading day; it is not a fix.
Assumptions that could move the date, stated rather than buried: a linear burn pace; other repos continuing at September's rate (unobservable from here); and the unresolved discrepancy in §2 below. What does not depend on any of them is the shape — a repo at 45% of the allowance cannot fix an account-level ceiling with an 8% trim.
The decision this implies belongs to the owner and has three weeks, not three days: accept a recurring ~6–8-trading-day blackout at each month's end; cut the account's burn by roughly half; or add a payment method and treat a few dollars of overage as the price of an unbroken forward clock.
2. /timing reports zero for every run, so the obvious detector data source is empty here
Anyone building the "how close is the month to the allowance" escalation this record asks for will reach first for the Actions usage API. It is empty on this account:
GET /repos/MaksimDan/thales/actions/runs/{id}/timing
→ HTTP 200, billable.UBUNTU.total_ms: 0
…for every one of the repository's 496 September runs, including runs that
demonstrably executed. Verified against run 35400417973 (Heartbeat Monitor,
2026-09-18, conclusion: success, 23 s wall clock): HTTP 200, total_ms: 0.
This is not a permissions or proxy artefact, and the distinction is worth keeping:
actions/cache/usage from the audit container returns 403 with an explicit
proxy message, while /timing returns a well-formed 200 with zeros. The
endpoint reports billed time, and on this account nothing is billed — every
minute sits inside the included allowance. It will stay zero for as long as the
account is on the free tier, which is the stated plan.
Measured that way, the method checks out against the one independent number we
have. Over created=2026-09-01..2026-09-30 the repository shows 496 runs / 557
jobs, and the 09-01…09-18 slice — the part of September that actually executed —
sums to 895 minutes, which is exactly the figure PR #185 arrived at by a
different route. That is 895 / 13 trading days = 68.8 min per trading day,
confirming the 65–70 in §2 of the 09-26 amendment and the ~1,450-minute month it
implies. Largest consumers over the window: Paper Trading 375 min, Skew Snapshot
146, Paper Trading (meanrev) 128, Fleet Digest 110, Auto-merge records 98, Paper
Trading (vrp) 82, Heartbeat Monitor 70.
Two cautions on those numbers, because the method has a known bias in both directions:
- It over-counts refused days. 09-21…09-28 adds 226 minutes by this method, and
every one of them is phantom: a refused job burns 3–13 s of wall clock and rounds
up to a minute, but
total_ms: 0and the annotation both say it never started, so nothing was billed. Restrict any such measurement to days that produced green runs. - It may over-count rounding. 895 rounded-up minutes against the billing page's $5.37 for thales is ~671 minutes at Linux's $0.008/min — a 224-minute gap. That gap is the right size to be the per-job rounding itself (557 jobs × ~0.4 min of waste each), which would mean the dollar figure is computed on raw seconds while the allowance is debited per rounded job. Which of the two the allowance actually charges is only visible on the billing page, and it changes the forecast date in §1 by a few days, not its direction.
A lever neither #185 nor #186 touches, worth naming because of that gap: this
repo runs ~43 jobs per trading day and every job costs a whole minute minimum, so a
seconds-long alert / email, beacon-relay or auto-merge-guard job is billed the same
as a one-minute one. Both queued PRs cut job duration (the redundant backup legs).
Cutting job count — collapsing the per-run alert job into its parent, or not firing
a guard workflow on pushes it will skip anyway — is an untouched and possibly larger
saving. Not proposed as work here; noted so the next burn pass looks at both axes.
So a burn detector must compute the number itself: sum per-job wall clock from
GET .../actions/runs/{id}/jobs (completed_at − started_at, each job rounded
up to the whole minute as GitHub bills it, skipped jobs excluded), over
?created=YYYY-MM-01..YYYY-MM-31. Two limits are structural and belong in
whatever gets built: it can only ever see this repository, so it reports a
share and never the account's headroom; and it needs Actions to be alive to run at
all, so like every in-band alarm here it goes quiet in the outage it predicts. The
useful form is therefore a forecast published while things are green — "at this
pace thales' share is spent on ~DATE" — not an alarm at the moment of refusal.
Not built here: it is new observability code, it cannot be validated while CI is refused, and a 12th open PR this week would compete with #185/#186 for the one scarce resource (a human merge). Recorded as a candidate for the research queue.
3. The 09-25 daily-audit escalation about October's stock selection was wrong
For the record, since that escalation went out and this file is where the outage's
claims live: the daily audit told the owner on 09-25 that a missed 2026-10-01 would
cost October's momentum stock selection outright, with the next chance 2026-11-02.
That is false. daily.py:_is_selection_day
(src/thales/execution/daily.py:463-518) keys selection off whether any run this
calendar month recorded selection_completed, not off a date; its docstring says
"a missed run can never skip a month (the next run that month selects)". Caught by
the weekly audit 2026-09-28 and re-verified independently by the daily audit the
same evening. The true cost of a delayed resumption is staleness — the book
rides the picks made 2026-09-01 until the first successful October run — not a
skipped month.