Two-replica audit drill

The S22 two-replica drill is a destructive test only of resources it creates under a randomized Compose project. It starts two Chorus processes against one real SurrealDB database and one real Garage bucket, gives each process a separate home volume, and binds every published port to loopback. It does not deploy or modify a production stack.

Prerequisites

A local Docker-API-compatible engine with Docker Compose 2.24.4 or newer (the override uses the Compose !reset and !override merge tags). Remote, SSH, and TCP contexts are rejected.
bun, curl, jq, openssl, and git
an already-built final Chorus root image in the same container engine
no production credentials in the shell; the drill generates private disposable credentials itself

Generate the source manifest first and bind its digest into the immutable image config. The drill regenerates the manifest and rejects a stale, unlabeled, or differently labeled image:

manifest="$(mktemp)"
bun run scripts/create-runtime-source-manifest.ts "$manifest"
manifest_sha="$(openssl dgst -sha256 -r "$manifest" | awk '{print $1}')"
docker build \
  --build-arg "RUNTIME_SOURCE_MANIFEST_SHA256=$manifest_sha" \
  --tag chorus-protocol:s22-final \
  .
rm -f "$manifest"

CHORUS_IMAGE=chorus-protocol:s22-final \
  ./scripts/compose-two-replica-drill.sh

Set COMPOSE_BIN when the approved Compose binary is not available as docker compose. Each run writes randomized, non-overwriting evidence and manifest names. S22_EVIDENCE_FILE, S22_SOURCE_MANIFEST_FILE, and S22_AUDIT_SOURCE_MANIFEST_FILE may select explicit archival paths. The script resolves CHORUS_IMAGE to its immutable local image ID before creating either replica, verifies the org.chorus-protocol.runtime-source-manifest-sha256 image label against the fresh manifest, sets pull_policy: never, and always passes --no-build when the replicas start.

What the drill proves

The drill fails unless all of these observable conditions hold:

Both replicas boot concurrently, run the same immutable image ID, become ready, and accept the same authenticated admin credential.
Atomic zero-config enrollment leaves the credential in exactly one of the two private home volumes. The key is moved through owner-only temporary files and never printed or placed in the evidence.
A signal written through one replica is readable and replayable over durable SSE through the other. Each capture is parsed as SSE, requires exactly one expected message, excludes prior signal IDs, and proves distinct, strictly advancing cursors.
While both Chorus containers remain live, the drill stops SurrealDB and requires both /health/ready probes to return HTTP 503 with {status:"not_ready", state:"ready", ready:false}. It restarts the same SurrealDB container, waits for database health, and requires both probes to recover to HTTP 200 without restarting either Chorus container. SurrealDB’s StartedAt must advance; both Chorus container IDs and StartedAt values must remain unchanged; and the pre-outage signal must still be readable through both replicas.
An artifact uploaded through one replica is byte-identical when downloaded through the other, proving both the SurrealDB record and Garage object are shared.
An authenticated SSE stream is observably active in the admin-only fixed-cardinality SSE registry when the first SIGTERM arrives, is closed by runtime-owned shutdown, and the process exits cleanly within the configured bound. A signal written while that replica is stopped is replayed after restart from the prior durable cursor.
With a separately admitted non-SSE request and a second authenticated SSE stream deliberately held open, a second SIGTERM forces the same owned shutdown within the bound and records trigger=second_signal. The machine proof records the SSE registry count and non-SSE admission count independently. A signal written while that replica is stopped is replayed after its restart.
Both recovered replicas are ready and authenticated, exactly one private home still contains the bootstrap config, previously committed signals and the Garage object remain intact, and complete logs from all four services contain none of the generated secrets.
Federation is explicitly disabled with its host key file fixed to /dev/null; all published ports remain loopback-only on a verified local engine.
Compose containers, volumes, and the network carrying the randomized project label are absent after cleanup. The same full-project reconciliation runs before the first container starts.

The default bound is 8 seconds and may be changed only within 1–30 seconds with S22_EXIT_BOUND_MS.

Evidence

A passing run writes:

.gsd/milestones/M001/slices/S22/S22-REPLICA-DRILL-EVIDENCE-s22-<run>.json

The committed JSON contract is schemas/audit/s22-replica-drill-evidence.schema.json. Beside each generated evidence record, S22-RUNTIME-SOURCE-MANIFEST-s22-<run>.sha256 contains a sorted SHA-256 manifest of the Docker runtime inputs (.dockerignore, Dockerfile, the root package/lock files, src/**, and packages/**). S22-AUDIT-SOURCE-MANIFEST-s22-<run>.sha256 independently pins the executable proof apparatus: workflows, Compose/config inputs, scripts, schemas, tests, operator docs, generated OpenAPI, packages, and immutable milestone plans. Generated results and mutable closure ledgers are excluded so the manifest has no self-reference. The script rejects any runtime or audit-source change during the run, and the validator re-hashes both archived manifests. Generated dependencies, build output, caches, environment files, logs, and Python bytecode are excluded.

The integrated gate requires at least two clean runs. Keep both randomized evidence/runtime-manifest/audit-manifest triples (or provide distinct -1 and -2 output paths), validate both, and require byte-identical source manifests plus equivalent proof invariants before closure.

Before reporting PASS, the script validates the final /3 record against the committed JSON Schema with AJV and then checks cross-field semantics such as image/source hash equality, replica image identity, cursor ordering, signal-ID parity, SurrealDB StartedAt advancement, unchanged Chorus container identity, and shutdown durations. proofs.surrealdb.restart records the exact outage/recovery readiness bodies, before/after container IDs and start times, and the replicas that reread the pre-outage signal. The evidence also records local engine identity, federation isolation, source state, Compose file hashes, replica ports, durable record/cursor identifiers, active-stream shutdown, recovery results, complete-log scanning, and verified project-scoped cleanup. It contains no API key, database password, Garage secret, or authorization header.

Archive the evidence beside the final source manifest and image attestation. A passing drill authorizes no rollout; production deployment remains a separate approval-gated action.