ORCHESTRATION POSTMORTEM: SIX HOURS, ZERO ROWS
Four terminals burned 6.5 hours and returned zero registered rows. This postmortem traces the retry loop and the worker that died, then lists the rules that now stop the waste.
Four busy terminals, 6.5 hours, and zero registered rows. On 2026-10-31, we dispatched two background workers to a mutation round. One champion strategy fed a ladder of candidate variants across four terminals, with about six hours allowed in the brief.
We stopped the round at 14:03. The scorecard showed roughly 250 requeue cycles and 13 full backtests that were never registered. The campaign card had not moved a single row. Six and a half hours of looping spent fleet compute and returned no registered row.
The run failed without crashing, and that was the dangerous part. There was no error page and no red alert, only busy machines and an empty card. We found no error to chase, so we reconstructed the failure from the log.
We read it against our broader three-phase testing protocol. The protocol separates clean, delayed, and robustness runs. This postmortem covers the plumbing around them: the queue, the journal check, and the worker that died.
#The brief asked too much of six hours
On paper, each worker had to find at least twelve qualifying candidates and climb from level one of the candidate ladder to level two, using four terminals inside roughly six hours.
Two workers split the fleet. One took two terminals and an explicit instruction to look for variants in the high-drawdown group. The other took the remaining two and subtler clues.
Both started at 06:38 without a pinned model. Each worker picked its own reasoning model at spawn time, so nobody downstream could tell which model produced which behavior.
Communication was sparse from the start. Worker B sent a heartbeat at 06:42 saying "investigating." Worker A sent one at 07:18 saying "implementing."
Neither worker sent another message. There was no completion, no escalation, and no further heartbeat. The logs only showed that silence later. It was the first failure signal, and we were not watching for it.
#The timeline shows one early win and a long retry storm
The driver, the script that queues each backtest, kept a 1,330-line log for the round. Every marker below is a logged event in that log or in the run inbox. None is inferred.
| Time | Event | What it meant |
|---|---|---|
| 06:38 | Both workers dispatched | Four terminals committed, ~6 h window opened |
| 06:42 | Heartbeat from worker B | "investigating" — the last message B ever sent |
| 07:13–07:30 | 13 runs banked raw on worker A's primary terminal | 12 unique candidates, one four-week window in early 2026 |
| 07:18 | Heartbeat from worker A | "implementing" — the last message A ever sent |
| 07:30:25 | Requeue storm begins | One unprovable cell starts looping to the front of the queue |
| 07:30–14:02 | ~250+ cycles, zero progress | Launch fails, full backtest runs, journal check fails, requeue — forever |
| 14:02:21 | Storm ends | Nothing registered; we stopped both workers |
| 14:03 | Aftermath | Three of four terminals empty; the primary holds leftover static charts |
All productive work finished by 07:30. We banked thirteen raw runs, covering twelve unique candidates, inside seventeen minutes on one terminal that did exactly what it was told.
Everything after 07:30:25 was the same cell requeued for 6.5 hours. We stopped both workers at 14:03. They returned a stopped state with an outcome of failed and a status of exited. No alert fired when either one stopped.
#A five-step requeue loop turned one unprovable cell into 6.5 hours of waste
The storm had one cell at its center, meaning one setfile tested on one time window. It was a deep-points variant of one of the first worker's candidates. Together they repeated a five-step loop with no exit condition.
| Step | What happened | What should have happened |
|---|---|---|
| 1 · Launch attempt | Fails instantly: the config path does not exist | Run the prepare chain first — inputs, then config |
| 2 · Requeue | Cell goes back to the front of the queue | Count the attempt; three and quarantine |
| 3 · Full backtest | ~75 s on the primary terminal, every cycle | Same — the compute was never the problem |
| 4 · Journal check | No config-load resolve for the setfile | Same — this check was correct and is the law |
| 5 · Verdict | "Not banked" → requeue to front | Mark journal-unproven, move on, escalate |
Our staging rule failed first. The backtest code was never the problem. Our two-step staging law requires a candidate's setfile to exist in every agent's sandbox, the private folder each worker runs from, under its exact configuration name.
The backtest ran on the primary terminal. The journal recorded no config-load resolve for that setfile, which is how we record that a configuration loaded. The law held for the window that produced the thirteen banked runs and failed for the storm cell. We never isolated the staging path and name mismatch inside the worker's own code, because we stopped the worker.
The right response was to quarantine an unprovable cell after a fixed number of attempts: leave it unbanked, move on, and escalate. Instead, the driver kept requeuing it, with no backoff, no fail counter, and no escalation path.
The terminals looked busy but produced nothing. One failed instantly. The other reran the same full backtest every cycle.
We call this a requeue storm. It has the same shape as the fleet-sweep "blocking read wedge": a retry path with no budget keeps coming back and spending machines.
The cost is what made this expensive. A full backtest on the primary terminal took roughly 75 seconds, and the storm spent hours repeating it for a cell that could never be banked.
Worker A's second terminal spent the evening failing instantly on a config path that never existed. Worker B's two terminals idled at zero. Four machines produced one useful window of seventeen minutes, and we had to escalate at 14:00 to find out. A counter that stopped after three attempts would have ended the loop before it consumed hours.
#One worker died silently while three of four terminals never launched a successful run
Worker B's side of the fleet died without a storm. Its agent terminal exited early, with its status marked "exited" before the stop order arrived.
Its only probe returned a run id of "0", our signal that no run launched, and opened zero charts. The worker's own ledger logged it as failed. It produced a tools directory, one candidate, and zero executed runs. A terminal that cannot launch is down, and we park it.
The prepare chain was the known cure: prepare inputs, prepare the configuration, then launch. A zero run id means the terminal is down and must be parked. Neither worker applied that rule.
We acknowledged two heartbeats, then stopped checking whether the workers were alive. We did not poll again until we escalated at about 14:00. By then both workers carried unverifiable liveness and a required-action flag, a message waiting for a response. We acted on neither.
No watchdog, autosync, or auditor daemons, the background services that watch a run, were running, and the health file had been stale for days. A worker can die silently. Our supervision has to catch it, and ours did not.
The same failure shape later surfaced in the data-integrity audit. A health check that only reads a registry can miss a worker that died outside it.
#The run produced 13 raw backtests but 0 registered results
The useful output was small, but it was not empty. Thirteen raw backtests existed, covering twelve unique candidates. None had crossed the registration door.
| Fact | Value | Status at 14:03 |
|---|---|---|
| Raw runs banked | 13 (12 unique candidates) | Exist as raw reports and log lines only |
| Window | Four weeks in early 2026 | Phase-one condition |
| Profit range | 1,355.88 – 2,195.98 | Unverified — not quotable yet |
| Trade counts | 3,192 – 3,203 | Band: green |
| Max equity drawdown | 8,598.40 on all 13 | Identical across all 13 — flagged for auditor eyes |
| Registration calls | 0 | The card still held only the two manual baseline cells |
Thirteen full backtests existed only as raw reports and log lines. The 1,330-line driver log contained zero registration calls. The campaign card still held only the two baseline rows we had entered by hand.
The storm cell's number never arrived, and the thirteen runs that did arrive never moved. All 13 carried the same 8,598.40 maximum equity drawdown. That may fit a shared baseline floor, but it was unproven, so we left it out of the record.
After the stop, three of the four terminals stood empty. The primary held leftover static charts from the killed loop: gold, fifteen-minute timeframe, one per chart. The next driver run had to close them one by one.
#Six root causes explain why busy terminals stayed empty
Six failures compounded. A bad cell met a broken staging path, three terminals that never launched a successful run, weak supervision, and a brief that let one stall crowd out every other worker.
- The loop had no attempt limit. One unprovable cell locked a worker for 6.5 hours. We had no max-attempts counter, no escalation path, and sent no completion message.
- Our staging law broke on the storm cell. The two-step setfile law held for one window and failed for another. A path and name mismatch inside the worker's own code stayed hidden because we stopped the worker.
- Three of four terminals never launched a successful run. One failed every attempt on a missing config path. One answered with run id "0". One never executed anything. The prepare-chain cure was written down and never applied.
- Worker B died silently. A structured background session with no pinned model went to "exited" status, and we noticed only hours later.
- No supervision from us. We acknowledged two heartbeats, then polled nothing until we escalated. Required actions sat unactioned, and the watchdog daemons were not running.
- The scope did not fit the window. Twelve qualifying candidates per worker, two ladder levels, four terminals in ~6 hours, with the launch protocol broken on three of four terminals and a design that let one bad cell starve the rest.
#Five circuit breakers now stop failed runs from burning the fleet
We adopted these recovery rules the same night, and they remain standing law:
- Never put the same cell and candidate back at the front of the queue more than 3 times. Then mark it journal-unproven or launch-failed and move on. Escalate through the inbox if a terminal is dead for more than 30 minutes.
- No banked result without proof. Before any launch on a fragile terminal, run the prepare chain and require a non-zero run id plus the config-load resolve in the journal. Run id "0" means the terminal is down. Park it.
- We poll worker liveness at least every 15 minutes while a run is live, and act on required-action flags instead of just acknowledging heartbeats.
- Register only what can be proven. Banked cells need journal proof and verified-report notes. The thirteen probe runs stay unregistered until the auditor proves them, including the identical-drawdown flag.
- Isolate failures per worker. One bad cell must never starve the other workers.
The successor round launched the next day with prove-first evidence, a fifteen-minute heartbeat, a three-attempt circuit breaker per cell, and park-on-zero for dead terminals. We left the scar in the operating record: the thirteen probe runs are still unregistered. The rebuilt driver now stops after three attempts instead of looping for six hours on a cell it cannot prove.
You can compare the current campaign state with the published DAAVILEFX results.
- Retry within a hard budget: the storm cell requeued roughly 250 times. The breaker now allows 3 attempts.
- Register only proven results: the 13 raw runs stay unregistered until the auditor proves them.
- Park a dead terminal before it consumes the round: run id "0" means the terminal is down.
Open the quantum dashboard to see the live evidence and the gates that now prevent a silent six-hour waste.
#Questions we get asked
#What happened during the 6.5-hour mutation round?
It was a requeue loop. Four terminals and two workers kept the round busy, but one unprovable cell was requeued roughly 250 times. The run ended with 13 raw backtests and 0 registered results.
#Why did the campaign card stay empty?
The journal could not prove the configuration loaded. The backtests ran, but our rules do not allow an unproven result to be registered.
#What stops the requeue loop now?
A three-attempt circuit breaker now stops it. We check worker liveness every 15 minutes and park a terminal that returns run id "0".
#Did the 13 raw runs become published results?
No. All 13 remain unregistered until the auditor proves them, including the identical 8,598.40 drawdown across every run.