Dispatcher daemon

chorus-dispatch-daemon is an execution-side process that runs outside the Chorus server. It polls one ring for ready tasks, claims each task through the lease protocol, creates a durable spawn intent, provisions a git worktree, and starts a CLI, Hermes, or remote worker. Chorus stores the dispatcher generation, task/run leases, checkouts, retry state, and signals; the daemon owns local process and worktree operations.

See the execution model for the task, run, and lease shapes.

Start and poll

CHORUS_API_KEY=... chorus-dispatch-daemon --workflow=.chorus/workflow.md

Flag	Default	Meaning
--workflow=<path>	.chorus/workflow.md	workflow document
--once	false	run one poll tick, preserve detached work, and exit
--vessel=<name>	ring name	override the dispatcher vessel name
--force	false	generation-fenced recovery or admin takeover
--grace-seconds=<n>	120	SIGINT/SIGTERM drain window
--force-kill	false	kill workers still live when the drain window ends

At boot the daemon parses the strict workflow schema, resolves environment references, calls whoami, resolves the configured ring, and registers (vessel, host). The deprecated chorus.identity field is informational only; the identity bound to CHORUS_API_KEY owns claims.

Each tick is ordered as follows:

1. Heartbeat the exact dispatcher generation. A stale process stops here before it can mutate subordinate work.
2. Reconcile exited or terminal workers.
3. Reconcile remote-unknown holds and durable retries, then poll and claim new work.
4. Heartbeat adopted live task leases.

Polling requests at most 100 tasks in poll.ready_states[0], filters by configured spawn kind, and enforces each kind’s concurrency limit. Task claiming is a storage-level CAS. Claiming sets execution/lease fields but does not rename the task’s workflow state.

--once leaves live work detached

--once exits after one tick. If it spawned or recovered live/unknown workers, it preserves their exact task leases and leaves the dispatcher registry active so a restart can recover them. If no subordinate work remains, the daemon shuts down its registry row normally.

Crash recovery and generation fencing

Registration is unique per (vessel, host). Two hosts may serve the same ring; task-claim CAS and per-kind concurrency are local to each daemon.

--force has two distinct paths:

• If the active row belongs to the same identity, Chorus atomically rotates that dispatcher’s generation and rebinds its live tasks and runs. The response contains the exact rebound task/run IDs. The daemon fetches only those rows and proves the new generation by task heartbeat before it probes, provisions, spawns, compensates, or retries anything.
• If the active row belongs to another identity, an admin force-takeover replaces it using the observed generation as a CAS fence and releases/reclaims the old generation’s work.

Both storage adapters fail closed if more than 1,000 tasks or 1,000 runs would be rebound; the generation is not rotated. The daemon rejects an oversized response too. A fresh registration performs no broad recovery scan.

Every newly claimed run gets a deterministic checkout ID before filesystem or process work. Its dispatcher_spawn metadata records the task/run/lease generation, deterministic handle, and one of these phases:

provisioning → spawn_requested → probing → running

Failure ends in compensated or quarantined:

• alive: adopt the exact handle. If workspace verification or phase persistence fails, keep heartbeating and counting the known-live worker with preservation-only cleanup.
• dead: generation-fenced compensation closes the run and persists retry state.
• unknown: quarantine remote work without compensation. The hold counts against concurrency until a task read proves, using the storage clock, that its exact lease fence expired or ownership changed. The daemon’s wall clock never releases that capacity.

Deterministic tmux/remote-tmux names let a restart probe the same worker after a crash between process creation and response persistence. A live worker is never killed merely because a later database write failed.

workflow.md

workflow.md is strict YAML frontmatter followed by the worker prompt template. Unknown keys are rejected.

Key	Default	Notes
chorus.endpoint	required	Chorus SDK URL
chorus.ring	required	task poll ring
poll.interval_ms	5000	delay between ticks
poll.ready_states	["ready"]	only the first value is polled
poll.terminal_states	["done","cancelled","archived"]	success cleanup and retry stop states
poll.lease_ttl_s	300	task lease and heartbeat TTL, 60–14,400 seconds
workspace.root	~/.chorus/work	worktree root
workspace.kind	worktree	clone and container parse but are not implemented
workspace.hooks.after_create	unset	shell hook after a new owned worktree
workspace.hooks.before_remove	unset	shell hook after ownership/cleanliness proof
agent.concurrency	{}	per spawn-kind cap
agent.default	1	fallback cap; zero disables that kind
agent.max_turns	10	prompt data only; the daemon does not enforce it
retry.max_attempts	3	consecutive failure budget, 1–100
retry.backoff.initial_ms	60000	first retry delay
retry.backoff.multiplier	5	exponential multiplier
retry.terminal_action	block	block, escalate, or cancel

Spawn-kind keys are cli:claude, cli:codex, cli:gemini, cli:opencode, hermes:<profile>, and remote:<host>. subagent and none cannot be spawned.

Durable retry and terminal action

Retry count is based on run.consecutive_failures, not the task’s lifetime attempt number. The daemon persists the count, configured maximum, relative retry delay, failure class, terminal action, and terminal-action receipt in the checkout. The checkout’s storage-minted updated_at anchors that delay, and checkout reads expose a storage-clock retry_ready projection. Local timers only decide when to check again; restart hydration and every claim attempt require retry_ready: true. Storage receives the same retry_limit, so InMemory and SurrealDB make the same lifecycle decision for workflow budgets above three.

On a terminal failure, cancel cancels the run/task lifecycle and block leaves the task in the gave-up path. escalate additionally emits an alert with a deterministic actor-scoped idempotency key derived from the run and lease generation. Signal, webhook-outbox, and durable stream-event creation are one storage operation. Replaying after either the signal response or the terminal-action receipt is lost returns the original signal without a duplicate alert, outbox row, stream position, or in-process publication. Reusing the key with different signal content returns 409. Restarts apply the terminal action recorded in the checkout, not a possibly changed workflow.md; an escalation is not acknowledged until its signal client successfully returns. A separate bounded, stable-cursor scan revisits blocked/degraded tasks, so a process crash after run compensation cannot hide an unapplied terminal action from the ordinary ready-task query. The final acknowledgement uses a dedicated generation-bound checkout command that verifies the already-terminal run/task disposition; it does not reopen live run authority or pass through the ordinary checkout transition graph.

Worktree ownership and reuse

Worktrees use <workspace.root>/<vessel>/<task> and branch task/<task>. A versioned .chorus/worktree-manifest.json binds the path to the repository common directory, canonical path, branch, base ref/commit, task, run, and lease generation.

An existing path is reused only when both the manifest and live git identity match. The same run/generation is adopted directly. A different generation may replace only an exact, clean, owned worktree; dirty, foreign, malformed, or unverifiable paths are quarantined. Cleanup runs before_remove only after ownership and cleanliness proof, then repeats both checks after the hook before its own removal. User changes and replaced paths are preserved.

Shutdown ownership

SIGINT/SIGTERM stops new claims while cleanup and heartbeats continue during the grace window. At the deadline:

• proven-dead workers may have their generation-fenced claims released;
• live or unverifiable workers keep their claims and keep the registry active for restart;
• --force-kill explicitly cancels remaining live workers before cleanup/release.

The same preservation rule applies to unexpected errors and --once. A second signal is logged and ignored while the first graceful shutdown is running.

Operational limits

• The dispatcher registry heartbeat is intentionally fail-fast. If its generation is stale, the process exits without subordinate mutation. Restart promptly with the same identity and --force while the registry/task leases are still recoverable.
• A remote probe may be unknown during an SSH/network failure. Chorus preserves ownership only until the current claim fence expires; operators should restore reachability or inspect the remote tmux session before that boundary.
• tmux has no portable historical exit code. A missing session is a proven-dead handle, while a task reaching a configured terminal workflow state is the authoritative success signal.
• Normal polling remains capped at 100 rows. Recovery is not subject to that page: it uses the exact bounded IDs returned by atomic generation rotation.