Observability

There is no Prometheus-compatible scrape endpoint or distributed tracer. What you get is a per-response meta block, one JSON log line per request, an admin-gated operational snapshot, and a handful of in-process loops that emit prefixed boot lines. This page is what an operator can actually see.

The server meta is the compatibility contract

serverIdentityMetaMiddleware runs after every handler and stamps a meta block onto any JSON response body (src/middleware/server-identity-meta.ts:15-44):

{
  "...": "your response payload",
  "meta": {
    "server_commit": "a1b2c3d4e5f6",
    "schema_version": "<current schema marker>",
    "request_id": "V1StGXR8_Z5jdHi6B-myT"
  }
}

• server_commit — the build commit, shortened to 12 chars. Resolved from the first set of CHORUS_BUILD_COMMIT / VERCEL_GIT_COMMIT_SHA / GITHUB_SHA / npm_package_gitHead, else "unknown" (src/version.ts:14-22). The Dockerfile wires it via an ARG GIT_SHA. This is your ground truth for which code is live — more reliable than any version string.
• schema_version — the build’s CHORUS_SCHEMA_VERSION application marker (src/version.ts:2). Treat the returned value as authoritative instead of hard-coding the example from this page.
• request_id — echoes the X-Request-Id header, or a fresh nanoid if absent (src/middleware/request-id.ts:9-14). It is also returned as a response header and is the only correlation key in the logs.

Why this is the contract

schema_version is a manually bumped application marker guarded by CI. Each newly applied SurrealQL migration records that marker and server_commit in the migration ledger, while startup independently validates the exact numbered/checksummed database prefix. The chorus-mcp client startup probe also compares client and server protocol/schema, exiting on a major mismatch and warning on a minor mismatch. Treat server_commit + schema_version as the client compatibility identity; inspect the migration ledger for database history.

The meta block is not universal

The middleware short-circuits on /signals/stream, /events/stream, and on any non-JSON content type (src/middleware/server-identity-meta.ts:18-19). SSE streams and binary/artifact responses carry no meta. It also re-serializes the body and drops content-length, so a downstream proxy keying off that header on JSON responses will not see one.

Request logging

loggerMiddleware emits exactly one JSON line per request to stdout (src/middleware/logger.ts:8-23):

{"timestamp":"2026-06-09T...","method":"POST","path":"/rpc","status":200,"latency_ms":12.3,"request_id":"V1StGXR8_Z5jdHi6B-myT"}

There is no log level, no structured error log, and no correlation beyond request_id (.audit lens-config-ops §6). Boot emits prefixed lines — [db], [schema], [reaper], [gc], [quota], [embedding], [artifacts], [mcp] — that tell you which subsystems came up.

Inbound rate admission exposes standard X-RateLimit-* headers and Retry-After; storage failure is distinguishable as 503 RATE_LIMIT_UNAVAILABLE. HMAC slot collisions increment a durable collision_count and emit a warning throttled to once per scope/slot/minute. Aggregate counters are not included in the operational snapshot.

Readiness and owned shutdown

GET /health/ready is the orchestration probe: it returns 200 with state ready only after startup has acquired every resource and bound the Bun listener. During startup, drain, or after closure it returns 503, Retry-After: 1, and the current runtime state. GET /health remains the liveness route; it is deliberately not a readiness or work-admission guarantee.

The first SIGTERM/SIGINT atomically changes the runtime to draining, rejects new work with SERVER_DRAINING, drains accepted HTTP and MCP requests plus SSE connections and background workers, then closes resources in reverse acquisition order. CHORUS_SHUTDOWN_TIMEOUT_MS (30 seconds by default) is one deadline for the entire operation. A second signal or deadline runs each registered force-abort hook for at most CHORUS_FORCE_ABORT_TIMEOUT_MS (one second by default), emits terminal diagnostics once, and then hard-exits nonzero even if an owned handle ignores its abort hook.

For the tracked Compose stack, CHORUS_STOP_GRACE_PERIOD uses whole seconds and must be strictly greater than CHORUS_SHUTDOWN_TIMEOUT_MS + CHORUS_FORCE_ABORT_TIMEOUT_MS; deployment preflight enforces the relationship before containers are changed.

The terminal [server] shutdown complete log carries structured JSON with state, trigger, duration_ms, unfinished, cleanup errors, and bounded per-component diagnostics. Alert on state=forced, any unfinished owner, or any cleanup error; those identify which accepted work or resource prevented a clean stop. Startup failure and interruption use the same owner graph, so only already-acquired resources are unwound.

Operational health snapshot

Admins can read GET /admin/health-metrics (or client.admin.healthMetrics()) for a bounded point-in-time snapshot. It includes database latency, uptime, delivery/dead-letter totals, embedding wake-hint depth, durable provider-work counts, 24-hour signal volume, lease protocol health, durable webhook health, and runtime_metrics. The runtime section has fixed-cardinality state/readiness, accepted-request and owned-component counts, terminal-shutdown and forced-shutdown counters, and the number of unfinished owners; it never labels a metric with an owner name. The lease section exposes fixed-cardinality counts for generation-bound active and reapable signals, tasks, runs, and dispatchers; legacy active authority; and the rollout phase/revision. webhook_metrics is also fixed-cardinality: outbox and delivery counts by state, ready/expired leases, and retained attempt events. provider_work_metrics reports the same fixed set of states for embeddings and LLM extraction. It never labels metrics by URL, work ID, memory ID, webhook ID, delivery ID, or identity.

object_storage_metrics is also fixed-cardinality. It reports all eight operation-state counts, global unresolved work, unresolved uploads, reservation and unresolved-ledger row counts, and the number of quarantined provider-inventory observations. It never exposes provider keys, object keys, namespaces, reference IDs, error text, endpoints, or credentials.

embedding_queue_depth counts only bounded in-process wake hints; it is useful for local pressure, but it is not the durable backlog. Use provider_work_metrics.embedding and provider_work_metrics.llm_extraction for restart- and replica-safe backlog, retry, dead-letter, and expired-lease alerting.

This is an on-demand JSON API, not a Prometheus endpoint. Poll it conservatively and alert on non-zero lease_metrics.legacy_active.total after enforcement, sustained growth in lease_metrics.generation_bound_reapable.total, growing webhook_metrics.outbox.pending, or non-zero expired webhook or provider-work leases. Alert immediately on provider dead letters and on sustained growth in pending + retry_wait. For object storage, alert immediately on any sustained dead_letter count, reservations != global_unresolved_uploads, or unresolved_ledger_rows != global_unresolved; investigate growth in quarantined inventory before any provider-side deletion. The public readiness body carries the same bounded runtime counts; alert on forced_shutdowns=1 or unfinished_owners>0 when collecting a terminal probe/log snapshot.

Background loops

These start in-process at boot via setInterval / croner — there is no external scheduler (src/index.ts:134-181,260-267):

Loop	Cadence	Env override	What it does
Lease reaper	30s	—	Releases expired signal-delivery leases back to pending, or to dead_letter after maxRetries (3); also ticks the dispatcher + run-attempt reapers (src/signals/reaper.ts:31-60)
Approval-gate reaper	60s	APPROVAL_REAPER_INTERVAL_SECONDS	Expires open approval gates
Webhook delivery engine	60s retry	—	Delivers signals + execution transitions to webhooks; retries at 5s / 30s / 5min, dead-letters after 3 attempts (src/webhooks/delivery.ts:33-43)
Embedding provider worker	event wake + 1s poll	—	Claims durable, generation-fenced embedding work; recovers missing embeddings and expired leases after restart
LLM extraction provider worker	event wake + 1s poll	—	Claims only explicitly created durable extraction intents; it does not launch an unapproved historical provider-cost scan
Memory admin job worker	event wake + 1s poll	—	Claims one generation-fenced import/export job, persists bounded page checkpoints, and prunes terminal history after seven days
Object-storage reconciler and inventory	60s	OBJECT_STORAGE_RECONCILE_INTERVAL_MS	Claims bounded guarded operations; verifies provider absence before settlement; commits signed-list inventory evidence and its durable cursor atomically; prunes only bounded terminal history after a complete inventory scan
Memory GC	5min	MEMORY_GC_INTERVAL_MINUTES	Applies decay, evicts below the floor
Artifact GC	15min	ARTIFACT_GC_INTERVAL_MINUTES	Only started when S3 is configured

The lease reaper caps each tick at 100 signals to avoid blocking the event loop, and logs + continues on a per-signal failure (src/signals/reaper.ts:29,54-59). The owned runtime closes admission and awaits active ticks before disconnecting storage.

GC visibility remains thin

The only GC accounting an operator can read is gc_last_run on the admin-gated GET /health/detailed. There is no per-loop metric or last-error surface. The reranker and query-router remain lazy, but embedding and LLM extraction now print explicit boot state and expose their durable database-owned work counts through the admin health snapshot. Object-storage reconciliation exposes durable aggregate state through object_storage_metrics; its per-sweep diagnostics remain runtime-owned rather than a high-cardinality public metric.

Federation v2 audit surface

Boot prints either [federation] V2 authority enabled for <public-origin> or an explicit disabled message. It never prints peer keys. Each admitted or durably rejected request creates linked federation_request and federation_receipt rows with peer, key generation, action, outcome, reason, result, and a hashed nonce; delivered signals carry dedicated federation peer/request/key provenance. Exact retries reuse those rows, so operators can distinguish acknowledgement loss from a second mutation. Raw nonces, plaintext HMAC keys, response bodies, and low-level transport causes are absent from this audit surface.

What there is not

Don’t build dashboards on what isn’t there

• No Prometheus/general scrape endpoint. GET /admin/health-metrics and GET /stats/overview return bounded JSON snapshots, not per-endpoint timing, histograms, or a Prometheus exposition format (.audit lens-config-ops §6).
• Audit coverage is handler-explicit, not uniform middleware. The unused auditAction() helper has zero callers; active sensitive handlers await their direct store.createAuditEvent(...) writes. Coverage therefore depends on each command declaring its audit event, rather than on a global interception guarantee. Read it through admin-gated GET /admin/audit.
• The listening log is not a durable readiness protocol. The listener is owned and bound before the log is emitted, but orchestration must still poll /health/ready so it observes the immediate transition to 503 during drain.

For the full env-var inventory and the additive-schema upgrade story, see Configuration and Database & migrations.