Durable memory administration jobs

Memory re-embed, export, and import are durable jobs. Admission writes the job, its fixed membership or immutable input chunks, and capacity ownership to SurrealDB before returning a job_id. Any healthy replica can resume an expired lease; a stale lease generation cannot settle replacement work.

All operations require administrator authority. The canonical command IDs are:

Action	REST	JSON-RPC	SDK
Start re-embed	POST /memory/admin/re-embed	memory.admin_reembed_start	client.admin.reembedMemories()
Re-embed status	GET /memory/admin/re-embed/:job_id	memory.admin_reembed_status	client.admin.getMemoryReembedStatus(id)
Start export	POST /memory/admin/export	memory.admin_export_start	client.admin.exportMemories()
Export status/page	GET /memory/admin/export/:job_id	memory.admin_export_status	client.admin.getMemoryExportStatus(id, options)
Start import	POST /memory/admin/import	memory.admin_import_start	client.admin.importMemories()
Import status	GET /memory/admin/import/:job_id	memory.admin_import_status	client.admin.getMemoryImportStatus(id)

The legacy slash-style RPC aliases for import/export starts remain compatibility shims. Prefer the dot-style command IDs for new integrations.

Re-embed

A start request has no body. Admission captures a database-clock upper bound and compound (created_at,id) cursor, fixing membership to the non-deleted memories visible at that point. Content revision still protects each provider result: if a memory changes before settlement, the old embedding attempt cannot overwrite the newer revision.

The status response reports processed, total, errors, and running, completed, or failed. The in-process queue is only a wake hint; durable job and provider-work rows are the authority after restart.

Export

Start with an optional namespaces array. Omitting it fixes the namespace membership observed at admission. A job exports at most 100,000 JSONL lines and 64 MiB. At most 100 namespace filters are accepted, each at most 512 UTF-8 bytes.

Snapshot membership is fixed at admission, but bodies and relations are read when each member is processed. The result is therefore a consistent membership export, not a historical byte-for-byte database snapshot.

After completion, page output with cursor, limit, and max_bytes:

• cursor is the prior download_cursor, starting at 0;
• limit defaults to 100 and is capped at 500 lines;
• max_bytes is capped at 1 MiB and may be reduced by the transport response ceiling;
• while has_more is true, pass the returned cursor to the next request.

If even the next single encoded line cannot fit, Chorus returns HTTP/RPC error 413 MEMORY_EXPORT_PAGE_TOO_SMALL with the required and available byte counts. Increase max_bytes or the negotiated response limit; do not skip the cursor.

Each export worker page reads at most 10 memories and 1,000 outgoing relations per memory. One output chunk is capped at 512 KiB of source JSONL, one line at 256 KiB, and hydration is capped at one response budget plus one boundary chunk.

Import

Submit a required lines array and optional mem0: true. One request accepts at most 1,000 lines, 8 MiB total, and 128 KiB per line. Input is persisted in immutable chunks of at most 25 lines and 512 KiB before the job is returned.

The worker imports memories first, then relations. Deterministic destination IDs and per-line mapping rows make replay idempotent. A 256-stripe database lock serializes namespace quota authority without making unrelated namespaces share one global lock. Invalid lines increment the error count; missing relation endpoints increment skipped_relations.

Capacity, retention, and recovery

At most 32 import/export jobs may be active and 256 job rows retained. Completed and failed job history is pruned after seven days in bounded pages. Re-embed jobs use their own bounded durable capacity and retention lane.

On process loss, leave the rows intact. A replacement replica reclaims the expired lease with a fresh generation and resumes from the committed checkpoint. Diagnose sustained running jobs by checking database readiness, worker boot logs, provider-work health, and expired leases. Never edit job counters, chunks, cursors, lock stripes, or generations by hand.

Migration 064 creates this authority. Back up before rollout. Because the migration and ledger advance commit atomically, a failed application is safe to correct and retry; after commit, prefer a forward code repair over deleting durable job state.