Object-storage reconciliation

Chorus never treats an object-provider response as database authority. Every application upload or delete first creates a durable SurrealDB guard; metadata, quota, unresolved capacity, retries, and provider inventory are then reconciled against that exact operation.

Invariants

• Upload bytes and unresolved capacity are reserved before provider I/O.
• A foreground PUT has an immutable generation and aborts before its five-minute handoff grace.
• Metadata settlement and reservation consumption are one database transaction.
• A delete keeps its metadata marker and charged bytes until provider absence is verified.
• Worker and admin settlements require the exact operation, owner, and lease generation.
• Provider inventory is observational. Unknown objects are quarantined; inventory never deletes.
• An incomplete inventory scan never advances retention or becomes absence evidence.
• Ambiguous uploads retain their tombstone and accounting even after observed absence.

Operator surfaces

All calls require administrator authority. The REST and canonical command names are:

Action	REST	Command / JSON-RPC	SDK
Aggregate health	GET /admin/health-metrics	admin.health_metrics	client.admin.healthMetrics()
Object health	GET /admin/object-storage/health	admin.object_storage.health	client.admin.objectStorageHealth()
Operation page	GET /admin/object-storage/operations	admin.object_storage.list	client.admin.objectStorageOperations(options)
Retry dead letter	POST /admin/object-storage/operations/:id/retry-cleanup	admin.object_storage.retry_cleanup	client.admin.retryObjectStorageCleanup(id)

Embedded MCP uses admin_object_storage_health, admin_object_storage_list, and admin_object_storage_retry_cleanup. See the transport parity matrix for standalone MCP and CLI names.

The list accepts up to eight unique states, an opaque returned after cursor, and limit 1–100 (default 50). Its summary deliberately omits provider/logical keys, reference IDs, request metadata, lease/write authority, endpoints, and credentials.

Read health first

object_storage_metrics and the dedicated health command return fixed counters only:

• all eight operation states;
• global_unresolved and global_unresolved_uploads;
• reservation and unresolved-ledger row counts;
• quarantined inventory observations.

Normally reservations == global_unresolved_uploads and unresolved_ledger_rows == global_unresolved. A mismatch, sustained dead-letter count, or growing quarantine needs investigation. Do not “fix” a mismatch by deleting a row.

Retry semantics

The retry input is only an operation ID. The server reads the operation kind and all provider, metadata, namespace, size, and lease authority itself.

For an upload dead letter, the command performs best-effort provider cleanup and returns action: upload_cleanup_observed plus one of provider_absent, provider_present, referenced, or provider_error. accounting_retained remains true for every result. A timed-out PUT can arrive after an absence observation, so the original physical generation is never reused and its tombstone is never absence-settled.

For an artifact, memory-body, or superseded-body delete, the command atomically transfers metadata and unresolved ownership to one deterministic protocol-v1 successor. It then performs bounded DELETE + HEAD under that successor’s fresh lease generation. Results distinguish completed, pending, and already-in-progress recovery. Repeating the source operation is idempotent; concurrent calls cannot create two accounting owners or two settlements.

Caller cancellation does not consume a cleanup observation. A provider timeout/error, stale metadata fence, live replacement lease, or failed settlement keeps durable ownership intact. Retry the returned operation when appropriate or let the ordinary reconciler resume it after lease expiry. Every admitted operator attempt writes a secret-free audit event.

Reconciler and inventory

Every configured interval, each replica attempts bounded claims. SurrealDB database time and fresh lease generations select one winner; stale workers cannot settle. The loop checks provider existence, deletes only an operation-owned unreferenced upload or exact pending delete, verifies absence, and then completes the matching metadata/quota transaction.

For providers with listing support, one durable scan row exists per prefix. A page’s reference, guard, or quarantine observations and its continuation cursor commit together in one transaction. Lease takeover retains the same scan ID and cursor but mints a fresh generation. Only a completed final page removes the scan row and permits inventory-retention work.

Terminal committed/completed operation history and stale inventory observations are pruned after 30 days in bounded pages. Recovered delete source tombstones are pruned only after the exact successor is completed, the source owns no unresolved ledger, and no metadata marker still names it. Ambiguous upload tombstones are retained.

Timeouts and capacity

General S3 requests default to 30 seconds and direct uploads to 270 seconds. Guarded application uploads use a stricter four-minute foreground deadline inside a five-minute grace. Deletes default to 20 seconds inside a 30-second grace. Reconciliation dead-letters after eight attempts.

Admission caps are 10,000 unresolved operations globally, 1,000 per namespace, 8,000 unresolved uploads globally, and 800 uploads per namespace. These are database-owned cross-replica counters. The normal reconciliation cadence is 60 seconds. See Limits for page, retention, and local-upload bounds.

Rollout and rollback

Back up both SurrealDB and the object store before migration 063. Stop or drain all old writers. The migration makes every pre-063 active operation an immutable protocol-v0 tombstone with accounting retained; protocol-v1 code is then required for new provider I/O.

After 063 commits, code-only rollback is unsafe. Old writers fail the required no-default protocol field before starting a PUT. Prefer forward repair. A full rollback requires coordinated restore of the pre-063 database and matching provider state, discarding all later writes.

Never manually delete an object, operation, metadata marker, quota row, unresolved row, inventory checkpoint, or tombstone. Preserve the evidence and use the typed recovery path.