Backup, restore & rollback

The tracked scripts back up and restore all persistent state owned by the bundled single-node Compose deployment:

• a portable SurrealQL export of namespace chorus, database protocol;
• Garage metadata and immutable object blocks;
• /home/bun/.chorus, which contains local Chorus bootstrap state;
• a non-secret manifest, the exact Compose/Garage configuration, and SHA-256 checksums.

Compose secret files, .env, and any TLS/proxy configuration are deliberately excluded. Store a protected copy of those separately. A Garage metadata snapshot is not useful with unrelated RPC/S3 credentials or a different deployment configuration.

Create and verify a backup

Run the normal deployment preflight first. SurrealDB and Garage must be healthy, and the Chorus container must already exist.

./scripts/deployment-preflight.sh
./scripts/compose-backup.sh /secure/off-host/chorus-$(date -u +%Y%m%dT%H%M%SZ)

The script unconditionally stops the existing Chorus container before export, including when Docker reports it between restart-loop attempts, so application writes cannot cross the Surreal/Garage snapshot boundary. It performs a logical surreal export, requires the OPTION IMPORT marker emitted by SurrealDB 3.x, then stops Garage before copying its metadata and data volumes. Garage explicitly warns against manually copying a running metadata database. Services that were running are restarted even if backup fails. The command returns nonzero if their original health cannot be restored; a completed backup payload is retained and identified separately from service recovery failure. The disposable Alpine helper receives only the exact named state volumes resolved from the service containers; Compose secret/config bind mounts are never inherited by the backup or restore helpers. An absent, duplicate, or unsafe volume name aborts before any writer is stopped.

The destination is created with owner-only permissions and promoted from a partial directory only after every file is present. When no destination is supplied, the script uses repository-local backups/; that directory is excluded from both Git and Docker build contexts, but it is not an off-host recovery copy. Move or replicate the completed snapshot to protected storage, then verify it again:

cd /secure/off-host/chorus-20260712T120000Z
sha256sum -c SHA256SUMS

Keep multiple dated generations and regularly test restoration on another host. A backup that has never been restored is only an assumption.

Restore

Restore replaces the complete contents of the dedicated Compose volumes. It refuses to run without a literal destructive confirmation, requires an exact checksum file set, rejects traversal plus link/device archive entries, verifies the pinned image family, and creates a fresh pre-restore safety backup by default.

./scripts/deployment-preflight.sh
./scripts/compose-restore.sh --yes-really-restore \
  /secure/off-host/chorus-20260712T120000Z

The target SURREAL_NS and SURREAL_DB must use the bounded identifier grammar and match the manifest. The script stops all three state owners, clears only their dedicated volume mount points, imports the SurrealQL export into an empty store, restores Garage and Chorus-home archives, and does not report success until all services are healthy and garage status succeeds. The one-shot Surreal import helper explicitly overlays the image-declared /data and /logs volumes with temporary memory filesystems, so it cannot allocate hidden anonymous volumes in the target engine.

If restore fails before state replacement, the script restores the original running-service state. If it fails after replacement begins, it requests all three state owners remain stopped and prints the safety-checkpoint path; keep the external proxy closed. Do not improvise writes into a half-restored instance. Fix the reported cause and retry the same idempotent full restore, or restore the automatically created backups/pre-restore-* checkpoint.

Upgrade and rollback sequence

Before any image, dependency, or migration upgrade:

1. Close external ingress and let owned work drain.
2. Run compose-backup.sh and copy the result plus secret/config escrow off-host.
3. Run the disposable restore drill or restore into staging using the candidate code.
4. Apply the upgrade and verify health, migration ledger, identity access, artifact download, stream continuity, and worker queues before reopening ingress.

If no incompatible migration committed, roll code/images back and restart. If a destructive or incompatible migration committed, code rollback alone is unsafe: restore the pre-upgrade backup with the old compatible image set. This loses every write after the checkpoint, so record the recovery point explicitly before proceeding.

Garage major-version upgrades must follow contiguous upstream migration guides. Its official procedure calls for an idle cluster and a stopped-node or built-in metadata snapshot before upgrade; do not assume a data-volume copy from a running node is valid.

Disposable proof

The repository includes an isolated destructive drill:

CHORUS_IMAGE=chorus-protocol:s22-final-<manifest-prefix> \
  ./scripts/compose-restore-drill.sh

The drill never builds or pulls Chorus. Before startup it resolves CHORUS_IMAGE to the local engine’s immutable image ID, generates a fresh runtime-source manifest, and requires the image’s org.chorus-protocol.runtime-source-manifest-sha256 label to match. A temporary Compose override removes build, sets pull_policy: never, and runs Chorus by immutable ID. Missing, stale, or remotely hosted images fail before any service starts.

It generates temporary secrets and a unique Compose project, writes randomized markers to SurrealDB, Garage, and Chorus home, backs up, mutates all three, restores, verifies the exact original values and absence of every post-backup value, then removes its containers, volumes, and network. Before writing markers it sends Chorus an explicit SIGTERM, requires a clean exit, starts it back to health, then performs and verifies a separate Compose restart. The project name combines the process ID with a random suffix, and before cleanup is armed the drill captures engine-wide volume and network inventories and refuses any existing project-labelled container, project-prefixed volume, or exact project network. It repeats that read-only collision check immediately before startup and requires the engine inventories to remain unchanged. Only then may cleanup reconcile Compose labels and the exact declared volume/network names created by the drill. Its interrupted-Compose fallback removes containers together with their anonymous volumes, and the successful cleanup proof requires every engine volume and network to match the pre-start baseline. It never targets the default or production Compose project.

After cleanup, the drill validates a strict machine-readable proof against schemas/audit/s22-restore-drill-evidence.schema.json plus cross-field semantics. The proof records the source commit and binds the byte-for-byte runtime-manifest hash to the image ID/label, Compose inputs and version, SIGTERM and restart timestamps, backup manifest and checksum identity, all three restored stores, and project cleanup. It records orchestrator_production_deployment_command_invoked: false, meaning the reviewed drill orchestrator did not invoke a production deployment command. The proof contains no raw generated credential, is published mode 0600, and is never overwritten. The randomized default is:

.gsd/milestones/M001/slices/S22/S22-RESTORE-DRILL-EVIDENCE-s22-restore-<random>.json

To select another new path, set S22_RESTORE_EVIDENCE_FILE. The command fails if that path already exists. A PASS line is printed only after JSON Schema validation, cross-field validation, owner-only publication, and verified cleanup all succeed.

The drill’s safety-backup bypass requires both the generated chorus_restore_drill_* Compose project name and an exact CHORUS_RESTORE_DISPOSABLE_PROJECT binding. A stray CHORUS_RESTORE_SKIP_SAFETY_BACKUP=true is rejected. Do not reproduce the bypass on a retained deployment.