Incident repair

When execution state gets stuck — a checkout never released, a run attempt orphaned by a dead worker, an approval gate nobody will vote on — you have a small set of repairs that actually work, and a larger set of admin commands that look like repairs but return 501. This page draws the line.

Generated from the code; claims point at file:line.

Repairs that work

These are real catalog commands on the member surface (HTTP / JSON-RPC / MCP). Each mutates state and emits an execution transition.

Operation	Command id	What it fixes
Release a checkout	`checkouts.release`	Returns a leaked checkout sandbox to the pool (`src/commands/handlers/tasks.ts:535-544`)
Repair primary delivery	`deliveries.promote_primary`	Re-points a run attempt’s primary delivery (`src/commands/handlers/tasks.ts:468-478`)
Cancel an approval gate	`approvals.cancel`	Cancels a pending gate; requester or admin only (`src/commands/handlers/approvals.ts:120-129`)

On MCP these are checkouts_release, deliveries_promote_primary, and approvals_cancel (src/mcp/task-tools.ts:250,266, src/mcp/approval-tools.ts:98).

Object-storage dead letters

Use the admin-gated operation list before retrying anything. The safe projection omits provider and logical keys, reference IDs, request metadata, lease authority, and provider configuration:

GET  /admin/object-storage/operations?states=dead_letter&limit=50
POST /admin/object-storage/operations/:id/retry-cleanup

The retry command branches on the server-owned operation kind; callers never submit an absence or settlement assertion.

An ambiguous upload is an externally uncertain write. Cleanup may attempt provider deletion and report provider_absent, provider_present, referenced, or provider_error, but it retains the immutable tombstone, quota reservation, and unresolved capacity in every case. Observed absence does not prove a timed-out PUT cannot arrive later.
A referenced artifact, memory-body, or superseded-body delete uses an exact type-aware successor guard. Metadata and unresolved ownership transfer atomically, provider delete/HEAD is bounded, and generation-fenced settlement releases accounting only after verified absence. Repeating the command returns the existing in-progress or completed recovery rather than creating another release owner.

Request cancellation stops provider work without consuming a cleanup observation. Provider errors, timeouts, reference drift, and stale fences leave metadata and accounting intact for a later retry or normal reconciliation. Every admitted operator attempt appends a secret-free audit event. Never delete provider objects, metadata markers, operation rows, reservations, or unresolved ledger rows manually.

Auto-recovery (the reaper)

A single in-process reaper ticks every 30s (src/index.ts:136) and runs four jobs in order (src/signals/reaper.ts:31-92):

Signal lease reaper — expired signal leases go back to pending, or to dead_letter after max retries (src/signals/reaper.ts:31-60).
Native task-lease release — expired execution-task leases are released (src/signals/reaper.ts:62-72).
Dispatcher registry reap — stale dispatchers are removed and their tasks released (src/dispatchers/reaper.ts:17-19).
Run-attempt reaper (#433) — orphaned run_attempt rows whose task claim no longer names them are transitioned heartbeat_stale (one-tick recovery window) then reclaimed / orphan_reaped (src/execution/run-reaper.ts:1-56).

Repairs that do NOT work — the 10 stubs

There is a second “RFC-291 operational repair surface” that was scaffolded ahead of its implementation (src/admin/rfc291-rollout.ts:85-104). These ten command names exist but every one returns 501 RFC291_ADMIN_NOT_IMPLEMENTED (src/errors/factory.ts:121) — after writing an audit event recording the attempt. They mutate nothing.

reclaim_run_attempt      reassign_task          unblock_task
cleanup_checkout         repair_primary_delivery clear_invalid_completion
cancel_orphan_gate       diagnose               reconcile_best_of_group
reset_failure_counter

Standing data-integrity caveats

From AUDIT-REPORT §4.6 / §4.1, still true at HEAD — know these before you trust a repair:

tasks.update cancellation does not cascade. cancelExecutionTask (with run-attempt cascade), archiveExecutionTask, and markCheckoutOrphaned are implemented but unreachable from any surface (§4.1). Cancelling a task via tasks.update leaves its run attempts.
attempt_number is assigned read-then-write; the unique index is the only race backstop (§4.6).
Best-of has no reconciliation despite schema and tooling implying it (§4.6).
Rehydrate caps scans at 100 signals — on a real instance it misses signals or duplicates tasks — and rehydrated/shim tasks point at a nonexistent ring:unassigned (§4.6).

For the broader scaffold story see The core loop.