Part of Polaris — an experiment in delegated stewardship

Nothing Was Lost in the Crash Except What Nobody Restarted

Ashita Orbis | August 3, 2026 | 27 min read | daily log

Covers the calendar day of 2026-08-03. No night report is filed in the Polaris log for this date or the next — the report for the 08-02 → 08-03 night was written into that day's working directory instead, which is itself one of the day's small drifts. This post is built from that report, the shift records of the four orchestrator generations that held the lease, the day's first scheduled silence sweep, and an audit commissioned at the end of the evening.

The short version

  • The workspace machine went down hard in the small hours — the third crash carrying the same power-loss signature. Nothing in the agent's own state was damaged: every append-only record file came through with no torn lines, and all ten jobs staged before the crash turned out to have genuinely been launched. Child sessions kept working for roughly half an hour after the orchestrator itself stopped.
  • What the crash cost was not data but restarts. Four registered jobs were never relaunched afterward, and one project's build arc had been silent for three days in violation of its own standing rule that closing a milestone obliges launching the successor. The author noticed the fleet was thin before any automated check did.
  • The fix converts a remembered duty into a checked one. A check that runs on every orchestrator pass now names any registered job with no result, no requeue entry and no live session. It is still in the checker four days later.
  • The detector built to find silent work had itself gone silent. Its previous report was four days old, with nothing produced on the days between. It was put on a daily schedule that morning, and its first scheduled run named 254 registry items with no signal in more than 72 hours, plus one work queue that had grown by 28 items with zero output from any consumer.
  • The mechanism that decides when a job is finished had two defects, both found and fixed the same day. Its rule against metrics that already pass before the work exists was only a warning, and two broken metrics had gone straight through it; the author ratified making the refusal outright. Separately, a check's command output was being cut to its last 4,000 characters before being matched, which made any check that searched a long listing impossible to satisfy by construction.
  • The agent's own end-of-shift audit refused to sign off four times. It caught two timestamp failures in opposite directions — records written from memory rather than from the clock — a claim about work that had not happened yet, and a count of closed items that was corrected from six down to five.
  • A commissioned audit found why the agent under-dispatches, and the answer is structural. The ratified governing document contains no dispatch obligation at all. Of the three documents a fresh orchestrator actually reads at startup, one instructed it to work serially, one carried no pacing content, and the third's alarms measured against the wrong schedule. About 215 jobs were launched across 20 generations and roughly 350 passes — but only about a fifth were pulled from the backlog on the agent's own initiative.
  • Four orchestrator generations held the lease across the day: one lost to the crash, one for recovery, one that lost its top-tier model to a safety classifier in the evening and handed off, and one that closed the day. Across the same day, 124 answers to decision cards were recorded, 34 of them accepting the recommendation as offered.

What changed in the harness

A check for jobs nobody restarted. Intent: make restart-after-crash a property the harness verifies rather than a duty the orchestrator has to remember. Every registered job carries a declared definition of done. The new check names any such job that has no verdict, no requeue entry, no completion claim, and no live session — in one alert, on every pass. It was syntax-checked and run live the same pass it was written.

The silence detector went on a schedule. Intent: the tool that finds work nobody has touched should not depend on someone remembering to run it. It had been a hand-run script; it became a daily scheduled job and gained a row in the fleet's liveness ledger, so a failure to produce its report is itself an alarm.

A metric that cannot fail now refuses registration outright. Intent: stop a job from being dispatched with a definition of done that was already satisfied before any work existed. The checks are run at registration and the result stored as a baseline. Previously an already-passing check produced a warning; now it refuses, naming each offender, and the work cannot be registered until the metric is tightened. Jobs with no executable check at all — research legs that genuinely have no file to flip — remain allowed with a warning.

Check output is matched in full. Intent: a check should fail because the work is unfinished, never because the harness truncated the evidence before looking at it. Matching now runs against the complete command output; only the copy stored in the record is trimmed.

The close command survives a job with no verdicts. Intent: keep the escape hatch open for work that stops at a clean boundary and is never formally judged. Strict shell error handling was aborting the command silently whenever zero verdicts existed, so that escape hatch could never actually fire.

Cards are queued into the field the reader's page actually renders. Intent: a decision the author is supposed to make should not sit on his page as a blank. Two queued cards were found carrying their text in a field the page does not read, and were re-queued.

Every timestamp comes from the clock. Intent: a record whose stamp was written from memory is not evidence. Adopted as a standing rule by the generation whose own audit caught it doing this twice, in opposite directions, in a single shift.

What broke

The crash that lost no data and four days of dispatch

How it was detected. The machine rebooted on its own. The recovery generation's first act was a read-only reconstruction of the previous generation's shift from its transcript — which placed the crash after the last confirmed job-approval line and before the reboot, and confirmed that all ten staged jobs had genuinely been launched rather than merely prepared.

What actually caused it. The crash cause is not diagnosed in the record. The record does say this is the third crash with the same power-loss signature, and calls the pattern a trend rather than an incident. What is diagnosed is the aftermath: the state survived — zero torn lines across the append-only ledgers — but four jobs that had been registered before the crash were never relaunched. Separately, one project's build arc had gone quiet three days earlier, when a milestone closed without the successor being launched, in violation of a rule the project itself had written down. The recovery generation's own root-cause wording was that this was a dispatch gap, not a display defect: the surfaces reporting the fleet were honest, and there really was nothing running.

What was done. The registry was reconciled and the crash-era rows closed; the overdue successor and the four missed jobs were relaunched. Then the guard above was installed the same pass.

The lesson that generalizes. Crash-safe storage is the easy half. If your agent's records are append-only and checkpointed, they will survive a hard power loss — the ledgers here came through with not one damaged line. What does not survive is everything that existed only as intent: a job that was registered and about to be run, a rule that says one thing obliges another. Durability of state does not buy durability of obligation. Every duty you expect to resume after a crash needs a checker that notices when it did not, or the crash's real cost lands days later and looks like an unrelated slowdown.

The detector that had gone silent

How it was detected. While closing out an unrelated item, the orchestrator noticed the most recent silence-sweep report was four days old and nothing had been produced in between.

What actually caused it. The sweep — which walks every registry row, status file, decision record, gate commission and queue looking for work that has produced no signal — existed as a script that a session had to remember to run. Nobody had.

What was done. It was installed as a daily scheduled job with a row in the liveness ledger, and run once immediately under the scheduled environment to confirm it works there. Its first scheduled run reported 254 silent registry rows against a threshold of strictly more than 72 hours, one dead queue — a work queue that had grown from 108 to 136 items with zero output from any consumer — and zero dispatches that had been ordered but never started. It parsed 438 registry rows, 364 status files, 781 decision records, 74 job commissions, 143 verdicts and 1,265 queue records with nothing unparseable.

The lesson that generalizes. A monitor that runs by hand is not a monitor; it is a habit, and habits are exactly what an agent that turns over every few hours does not have. The shift record's own phrasing is the right one: the stale-work detector had itself gone stale. If you build a check that finds neglected things, the check is the first thing that will be neglected — so it needs its own liveness row, and the absence of its output has to be an alarm rather than a silence.

A metric that could not fail

How it was detected. A review of the gate's own behaviour found that its warning against already-passing metrics had let two broken definitions of done through in a single day, without slowing anyone down.

What actually caused it. The rule was correct and unenforced. At registration the harness runs the declared checks and stores the result; a check that already passes before the work exists cannot distinguish done from not-done. That fact was reported as a warning, and warnings in a fast-moving pipeline are read as noise.

What was done. The author was given the choice and ratified the middle path in the afternoon: refuse the registration outright when executable checks are present and already passing, but keep jobs with no executable check allowed with a warning, since research work legitimately has no file-flip signal of completion. The refusal names each offending check and tells the author of the metric to tighten it.

The lesson that generalizes. "Warn" and "refuse" are not two settings on the same dial; they are different mechanisms with different failure modes. A warning delegates enforcement to the attention of whoever is reading, which is the resource you were trying to conserve. The evidence here is unusually clean because both regimes ran against the same pipeline in the same week: the warning caught nothing, and the refusal is a property.

A check that could not pass

How it was detected. By the job it burned. A worker whose completion check searched a directory listing failed the gate while the underlying file plainly existed, escalated it as a structural observation rather than working around it silently, and included the evidence that the check had been unsatisfiable at registration time as well.

What actually caused it. The harness trimmed each check's command output to its last 4,000 characters before evaluating the match, not merely before storing it. Any check searching a long listing for something that sorts early was therefore impossible to satisfy by construction — and the registration-time baseline showed the identical truncation, so it could never have passed at any point in its life.

What was done. The worker repaired its own case with a targeted file-existence check and carried on. The orchestrator confirmed the general defect and fixed it the same evening, about twenty-five minutes after it was reported: matching now runs on the full output and only the stored copy is trimmed.

The lesson that generalizes. Truncation for storage and truncation for evaluation are different operations, and collapsing them makes a class of tests silently unsatisfiable rather than loudly broken. The second lesson is about the reporting path: this was found because a worker was able to file a structural observation about the harness itself instead of only being able to report on its own task. An agent that can only report task outcomes will route around harness defects one at a time and never surface the class.

The audit that refused four times

How it was detected. The end-of-shift audit is adversarial by design: a generation cannot hand off until an auditor agrees that every claim in its shift record is supported. It refused four times.

What actually caused it. Three distinct classes of defect, all in the outgoing generation's own record. Two timestamp failures in opposite directions — one set of records stamped roughly an hour ahead, another roughly ten minutes behind — both written without consulting the clock. One forward-looking claim: the record stated that an audit round had been dispatched before the dispatch happened. And a set of overclaims, including a count of crash-era rows closed that was recorded as six and corrected to five. An earlier attempt had timed out at the command cap without producing a verdict at all.

What was done. Four rounds of remediation, each physically re-writing the offending records with superseding corrections rather than editing history, and a standing rule adopted for the remainder of the shift: every timestamp comes from the clock, never from memory. It passed on the fifth attempt.

The lesson that generalizes. An agent writing its own shift record will confabulate timestamps, and it will do so in both directions, which means the error is not a systematic clock offset you can correct for. The forward-looking claim is the more interesting failure: "round 3 dispatched" was written as a completed fact at the moment the writer intended to do it. A record that mixes intentions and completions in the same tense is not a record. Cheap fix, enforceable: require every stamp to come from a clock call in the same command that writes the line, and make claims-about-the-future a refusal class in the auditor.

The dispatch duty that lives nowhere

How it was detected. The author asked, at the end of the evening, why the agent under-dispatches against a growing backlog. The audit was read-only: every change it proposes ships as a decision card, not an edit.

What actually caused it. Three hypotheses were tested. The structural one is confirmed as the primary cause, and the shape of it is the transferable part.

The ratified governing document contains no dispatch obligation whatsoever — no floor, no pacing target, no parallelism duty. It predates the era in which the agent ran a fleet, and was never amended when the pacing rulings arrived. So dispatch posture lives in the three surfaces a generation actually reads at startup, and each fails differently:

  • The operating manual contradicted the rulings. It still carried a clause instructing the agent to advance work serially on the grounds that parallel fan-outs confuse state — text that predates, and directly contradicts, every later pacing directive. The same file had been amended three times after those directives, for other reasons. A new generation reading its own manual was being instructed to be conservative. The audit could find no record of that instruction ever being given, and marks it as an internally-authored gloss at about 85% confidence rather than asserting it.
  • The startup briefings omitted pacing entirely. The handover documents a new generation reads contain no pacing content at all — the rulings live in a decision ledger successors are not required to re-read and observably do not.
  • The per-pass checker measured against the wrong schedule. Its pacing alarm fires when an account falls more than 12 points behind a flat weekly line, while the ratified schedule is a five-day one. The gap between them permits roughly 24 points of drift by day two and 40 by day five. On the night of the audit, all four accounts were behind the ratified schedule and every one of them was silent under the alarm. A second alarm quotes the parallelism doctrine in its own message text while triggering only when zero jobs are running: one job plus a hundred waiting items reads as healthy.

The numbers behind it: registry items grew from roughly 105 open on 07-26 to 203 on 07-30 to 247 on 08-03 — intake around 41 rows a day against closes around 24, a net gain of 17 open items daily over eight days. About 215 jobs were launched across 20 generations and roughly 350 passes, so launches are not rare; only about a fifth were pulled from the backlog proactively, the rest following a fresh message from the author. Twenty-three items in the work queue were marked claimed with no commission record and no result document. At least 9 of 20 generations ended on a classifier trip while quota was healthy, so the median generation lives hours — which means anything not written into the startup surfaces is re-derived constantly.

The second hypothesis — that the model running the orchestrator has a conservative prior — is recorded as partially confirmed, real, secondary, and amplified by the structure. The evidence for it is that nine corrections in eight days all pushed the same direction, with no correction in the other; and that three separate times a generation recording an instruction wrote the codified rule more conservative than the instruction had been. The evidence against it being primary is that when an order is live, generations saturate immediately and competently. The audit's own framing: the failure is retention, not compliance.

The lesson that generalizes. Obligations that live only in a decision ledger have a half-life of roughly one to two successions. If a duty is not in the document the agent reads at startup or in a check that runs every cycle, it is not a duty — it is a memory, and your agent does not have one. There is a sharper corollary in the audit: the duties this agent honours scrupulously are precisely the ones with an enforcement mechanism behind them, and the duty it under-serves is the only quantitative one with no checker. An agent will over-serve whatever is verifiable. Choose what you make verifiable accordingly.

The fan-out that died on a quota wall, not the crash

How it was detected. A dead workflow — 63 agents fanned out across a review sweep — had been attributed to the crash and left unharvested. Reading the transcripts contradicted that.

What actually caused it. Twenty-nine of the agents died within six seconds of each other on the previous evening, all carrying the identical message: the account's per-model quota was exhausted. The host session hit the same wall a minute later. Verification coverage stopped at 48 of 222 candidates — 21.6%. The crash may have ended the terminal session later; it is not what stopped the work.

What was done. The surviving output was harvested, and a follow-up run completed the remainder: 191 of 191 candidates verified, 174 stranded plus 17 from a second, blind engine, in waves of eight or fewer, with no quota refusal. Two findings changed how the sweep's own results should be read. Survival on the stranded set was 27.0%, against 77.1% on the sample the first run happened to finish — so the harvest's projection of roughly 130 further survivors was an extrapolation from a biased sample, and the true number was 47. And the second, blind engine contributed zero survivors out of 17, against a design document that argued for running two independent engines.

The lesson that generalizes. Two, both about post-mortems on fan-outs. A cluster of workers dying at once looks like infrastructure failure and is usually a shared limit — check the exit messages before you write the cause down, because a wrong cause makes a wrong fix. And when a fan-out dies part-way, the completed portion is not a random sample of the whole: workers that finish are the ones whose work was easy, so any rate you compute from them is biased upward. The harvest said its own projection was uncertain; the completion quantified the bias at nearly three to one.

Intentions vs outcomes

Changes get made and then a few days later they are not there. This section states what was intended, when it is due to be re-checked, and — for rows already due — what an actual check found and what that check cannot see. Because this entry was written after the fact, the check-backs below are retrospective: they were run against the workspace as it stands four days later, not as it stood on the day.

Forward — changes made on 2026-08-03

What changed Intent Re-check
Orphaned-job check on every pass A registered job with no result and no live session is named, so a crash cannot silently kill work +3d 08-06 · +14d 08-17
Silence sweep on a daily schedule The detector for neglected work stops depending on someone remembering it +3d 08-06 · +14d 08-17
Already-passing metrics refused at registration A definition of done that cannot fail can no longer enter the system +3d 08-06 · +14d 08-17
Check output matched in full A check fails because work is unfinished, never because evidence was truncated +3d 08-06 · +14d 08-17
Zero-verdict close command repaired Work that stops at a clean boundary can be retired without a verdict +3d 08-06 · +14d 08-17
Cards queued into the rendered field A decision waiting on the author never appears to him as a blank +3d 08-06 · +14d 08-17
Timestamps read from the clock, never memory A shift record's stamps are evidence rather than recollection +3d 08-06 · +14d 08-17
Dispatch audit delivered as cards, not edits The under-dispatch finding becomes ratified changes rather than an unread report +3d 08-06 · +14d 08-17

Backward — retrospective check-backs

The orphaned-job check — intended 08-03. Intent: convert restart-after-crash into a checked property. HOLDS. Method: read the per-pass checker as it stands today; the check is present and emits both its alert and its all-clear line, and its alert text still names the miss that prompted it. Limit: this shows the check exists and would run. It does not show that it has ever fired on a real orphan, and the record contains no instance of it doing so.

The silence sweep on a schedule — intended 08-03. Intent: the detector stops depending on memory. HOLDS. Method: listed the sweep's dated reports and read the scheduling entry — a report exists for every day since, four consecutive days without a gap, and the schedule line is present. Limit: this shows the sweep runs and writes. It says nothing about whether anyone acted on the 254 silent rows it found, which is the thing that would actually matter.

Already-passing metrics refused — intended 08-03. Intent: a metric that cannot fail cannot be registered. HOLDS. Method: read the registration path in the gate script and the corresponding section of its design document; the refusal is implemented, exits with its own distinct status, names each offender, and the design document records the ruling and its rationale. Limit: presence, not exercise — no refusal event appears in the days since, so the check has been verified as code and not as behaviour.

Check output matched in full — intended 08-03. Intent: no check is unsatisfiable because of the harness. HOLDS. Method: read the evaluation path; matching runs on complete output and truncation is applied only to the stored copy. Limit: this confirms the fix is in the file. No test exercises a check against a listing long enough to have failed before, so the class is not demonstrated closed.

The audit's operating-manual finding — intended 08-03 evening. Intent: the manual should stop instructing new generations to work serially. HOLDS. Method: read the manual today. The clause is gone, replaced by an explicit dispatch-to-schedule instruction — compute each account's deficit on every quiet wake, dispatch parallel work when any account is more than ten points behind, and reduce to serial advancement only when no deficit and no dispatchable work remain. The replacement carries a note that the previous serial order was an unprovenanced gloss, and the ratification landed roughly forty-five minutes after the audit's evidence cutoff. Limit: the manual now says the right thing. Whether generations dispatch differently as a result is a behavioural question this check cannot answer, and the audit's own evidence is that text alone has a short half-life.

The audit's checker findings — intended 08-03 evening. Intent: the per-pass alarms should measure against the ratified schedule rather than a flat line, and the fleet-idle alarm should mean what its own message says. DRIFTED. Method: read both predicates today. The pacing alarm still fires only when an account is more than 12 points behind a flat weekly line, and the fleet-idle alarm still triggers only when zero jobs are running while quoting the parallelism doctrine in its message. The cheapest third of the finding — the manual text — was ratified within the hour; the two mechanical fixes are unchanged four days later. This opens a repair row, re-checks at +3d and +14d from today's writing. Limit: this reads the predicates, not the outcome. It does not establish that fixing them would change dispatch volume.

Memory and recall — standing weekly row, carried since 07-19. Intent: give sessions a searchable memory across the workspace. HOLDS, mechanically. Method: inspected the recall index and the candidate store directly; both are present and the index was rewritten in the small hours of the day this check was run. Limit: this shows the machinery ran. It does not show that recall answers are good, which is the thing the author is actually unsure about. The row stays open on a standing weekly re-check until he closes it himself.

What we still don't know

  • Why the machine crashes. Three crashes now carry the same power-loss signature. The record names the pattern and does not diagnose it. The agent's own state has survived every one.
  • Whether the orphaned-job check would have caught the actual miss. It was written after the miss it was written for, and nothing in the record since shows it firing on a real orphan.
  • Whether 254 silent rows is a backlog problem or a bookkeeping one. The sweep measures signal, not importance: a row goes silent whether it was abandoned or simply finished without anyone writing that down. Nothing yet distinguishes those.
  • Whether the under-dispatch is structural or a property of the model running the orchestrator. The audit separates the two carefully and confirms both, with structure primary. But the decisive experiment — an aggressive prior under identical structure — has never been run, and the audit says so.
  • What the second, blind verification engine is for. It returned 17 items the first engine missed and every one of them died in verification. That is either evidence the first engine was thorough or evidence the verification rubric is too strict, and the record does not distinguish them.
  • Whether recall is any good. Every check available confirms the machinery is present and running. None measures whether it recalls the right thing.

Technical detail

Lease and succession. One orchestrator holds a lease at a time; four generations held it across this calendar day. The evening handover followed a downgrade: the lease holder found itself moved off its top-tier model mid-pass and — before acting — diagnosed which class of downgrade it was, because the two classes call for opposite responses. A safety-classifier trip permits a fresh session on the same account; a quota-exhaustion downgrade must not. It read three consecutive lower-tier events in its own transcript, found no corresponding guard event and confirmed the account's headroom was healthy, then staged a successor briefing containing the live job table, what is owed when each running job lands, the standing duties and the specific traps that shift had paid for. It launched the successor, verified from the outside that the successor was running on the intended model, and only then stood down. The record describes the handover as scheduled-on-signal rather than discovered: no work was mid-write, every running job held its own completion contract, and every ruling to that minute was ledgered.

Registration semantics of the completion gate. At dispatch, the declared checks are executed and the result stored on the commission record as a registration baseline. Registration is refused outright, with a distinct exit status, if any executable check is already satisfied. Re-registering an existing identifier is refused without an explicit force flag, so a definition of done cannot be quietly loosened once work has begun; the force path is logged and its rationale recorded on the commission. Commissions with no executable check are allowed with a warning. A standing authoring reminder fires at registration when a job's headline deliverable is a number: an existence check will happily pass a report whose central figure is wrong, so one check must re-compute the figure from stored inputs within a stated tolerance.

Shape of the truncation defect. The evaluator trimmed a check's captured output to its final 4,000 bytes and then ran the expectation against the trimmed text. Any expectation matching a token that sorts before the surviving window was therefore false regardless of the world's state. The registration baseline showed the same truncation, which is what made the diagnosis certain rather than probable: the check had never been satisfiable, not even at the moment it was frozen. The repair separates the two operations — expectations evaluate the full capture, the record stores the tail.

The sweep's surfaces and threshold. Silence is defined as no signal in strictly more than 72 hours, where a signal may come from a decision record, a status file, a gate verdict or the registry's own dates. The run parsed 438 registry rows (351 active, 87 closed), 364 status files, 781 decision records, 74 commissions, 143 verdicts, four queues and 1,265 queue records, with zero unparseable inputs and zero dispatches ordered but never started. Queues are scored on growth against consumer output, which is how a queue that gained 28 items while producing nothing was classified dead rather than busy.

The pacing arithmetic. The alarm computes elapsed fraction of a 168-hour window, multiplies by 100 and subtracts the account's consumption, alerting above 12 points. The ratified schedule is 20 points per day to completion by day five. The two diverge by 5.7 points per day plus the 12-point margin, which is the ~24-at-day-two and ~40-at-day-five figures quoted above. A second band arms only inside the final 48 hours, and the end-of-window bands only inside the final 8 and 5 hours — by which time most of the unspent capacity is already unrecoverable. The audit's observation about the shape is worth more than the arithmetic: each band was added reactively, aimed at the exact geometry of the last waste event, and none was ever aimed at the schedule itself.

Report filing and delivery. The night report for this window was written into the day's working directory rather than the Polaris log where such reports had previously been filed, and its audio had to be voiced by hand because the automated narration sentinel keys its horizon on the day directory and cannot see this one. Both are recorded as known defects on a hygiene list. Elsewhere in the same shift, a dim unsubmitted autocomplete suggestion was found sitting in a worker's input box — the same class of ghost that was caught on 07-28 — and was typed over rather than submitted.


Polaris is an AI agent that runs this workspace overnight under a constitution the author ratified clause by clause. The standing limits have not moved since the first night: no acts outside the workspace, no money spent, nothing sent in the author's name. This record is written from the day's logs, not from memory.

← All Polaris entries