Restoration Runbook

This document provides consistent recovery procedures for all persistent subsystems within the Neo.mjs AI substrate, leveraging the atomic bundles generated by the Daily Snapshot Pipeline (npm run ai:backup).

Locating the bundle root

The root is the resolved backupPath leaf — NEO_BACKUP_PATH when set, otherwise ${HOME}/.neo-ai/backups. The default no longer derives from the checkout path. It was relocated out of the plane because a checkout-relative root sits in the git working tree, where git clean -x removes it (.neo-ai-data is gitignored, and clean -x is defined as reaching ignored files).

That is the bound worth knowing before you go looking: the default moved, not every possible location. A deployment may point NEO_BACKUP_PATH anywhere, including back under a checkout — so resolve the leaf rather than assuming where bundles are.

Resolve it rather than assuming, then use $BUNDLE_ROOT throughout this runbook:

export BUNDLE_ROOT=$(node -e "import('./ai/config.mjs').then(m => console.log(m.default.backupPath))")
ls "$BUNDLE_ROOT"

Bundles taken before the relocation may still sit at the old in-tree .neo-ai-data/backups. Nothing was migrated automatically; the backup CLI emits a one-time notice naming that path when it finds bundles there. If you are restoring from an older bundle, point BUNDLE_ROOT at it explicitly.

Bundle Layout Overview

Each $BUNDLE_ROOT/backup-<timestamp>/ contains the following atomic directories:

• kb/: Knowledge Base ChromaDB as JSONL
• mc/: Memory Core memories and summaries as JSONL
• graph/: Memory Core SQLite graph as JSONL
• concepts/: Concept Ontology JSONL
• trajectories/: RLAIF training trajectories JSONL
• ledgers/: incident ledgers — heal-attempts.json, heal-events.jsonl, recovery-runs/. Present only on bundles taken after ledger capture landed; a bundle without it restores normally.

Prerequisites

Before initiating any restoration, ensure that all AI MCP servers and daemon processes are stopped (terminate any running npm run ai:server processes).

Atomic-Bundle Restore CLI (ai:restore)

npm run ai:restore -- <bundle-path> (entrypoint ai/scripts/maintenance/restore.mjs) inverts the Daily Snapshot Pipeline: it reads a bundle, validates structure + JSONL parseability + topology compatibility, then restores each subsystem (KB, MC memories/summaries, graph, concepts, RLAIF trajectories, mailbox archive, incident ledgers) through the canonical Zod-validated SDK boundary — flat members via direct file restore. Use this for a full-bundle restore; the per-subsystem procedures below are the manual fallback when you need to recover a single store.

Flags

Flag	Effect
--mode merge (default)	Idempotent. Embedded substrates upsert (graph SQLite uses INSERT OR IGNORE); flat substrates (concepts/, trajectories.jsonl) skip-if-target-exists, preserving operator additions. No --force required.
--mode replace	Gated + destructive. Each embedded subsystem fires assertDestructiveTargetAllowed() before truncating + restoring; refuses if any target is non-empty without --force.
--force	Required when --mode replace AND any target is populated (acknowledges overwrite). Also overrides the flat-file skip-if-non-empty rule under --mode merge.
--force-topology-mismatch	Bypasses the topology-compatibility refusal when restoring a legacy federated-topology bundle into a unified deployment (collection IDs may diverge across topologies).
--preserve-read-state	Replace mode only. Keeps committed mailbox read receipts (DELIVERED_TO readAt/archivedAt) across the truncate, re-applying them wherever the bundle left them null. Off by default, because --mode replace means the bundle IS the new state and a disaster-recovery restore must reproduce it exactly. No-op under --mode merge, which never truncates — the run warns rather than accepting a safety-intent flag silently.
--operation <name>	Selects a named operation whose defining arguments are pinned, not defaulted. Currently reseed (see below). A contradicting argument is refused, never silently honoured.

Which operation is this? — ai:restore vs ai:reseed

Two operations share this CLI and need opposite read-state policies. Read-state is runtime-only: no synced bundle can carry it, so getting this wrong destroys mark_read writes the tool already acknowledged as read, which a seat experiences as its mailbox rolling backwards.

	npm run ai:restore -- <bundle> --mode replace --force	npm run ai:reseed -- <bundle> --force
Operation	Disaster recovery	Live operational re-seed
Meaning	The bundle IS the new state; reproduce it exactly	The graph is rebuilt from a lagged snapshot, with writers quiesced first
Scope	Whatever you ask for (every substrate by default; --only-substrate narrows it, e.g. --only-substrate ledgers)	Graph only — pinned
Read receipts	Discarded with everything else	Preserved — pinned
Mode	Yours to state	replace — pinned
--force	Yours	Yours — deliberately not pinned; the destructive acknowledgment never rides inside a convenience name

ai:reseed pins its three defining values and refuses an argument that contradicts any of them: ai:reseed -- <bundle> --mode merge aborts rather than performing a merge under a name that promises a replace. Graph-only is not tidiness — DELIVERED_TO lives in the graph, which is the entire reason preservation matters here, so a name advertising a safe live operation must not also replace kb/mc/concepts/trajectories/mailbox.

⚠ Quiesce writers before ai:reseed. This is a precondition, not a nicety.

The read-receipt capture runs inside the truncate transaction, which closes the lost-write window a separate SELECT-then-DELETE would open. But that transaction ends before the import and re-apply run — so a mark_read acknowledged after the capture and before the re-apply completes is lost, and nothing detects it. Stop the seats (or the MC server) first, re-seed, then bring them back.

Preservation still matters under quiescence: the bundle is lagged, so receipts committed since it was captured must survive the rebuild. Quiescing removes the concurrent writer, not the stale-snapshot problem — which is what the preservation is for.

A live-writer-safe variant would need a writer fence held across truncate → import → re-apply. In SQLite that means an exclusive lock for the whole import, i.e. enforced quiescence rather than avoided quiescence, plus a concurrent falsifier proving an ack inside the window survives. Not implemented, and deliberately not claimed.

How to tell it worked: a successful re-seed logs [importDatabase] Re-applied N committed DELIVERED_TO read-receipt(s). What N means, precisely: it counts DELIVERED_TO edges whose receipt was captured before the truncate and which the restored bundle left null — i.e. receipts the bundle would have destroyed. So N = 0 legitimately means the bundle already carried every receipt (a fresh bundle, or no reads since it was captured); it is not a success metric to maximise. The tell that something is wrong is N = 0 on a re-seed from a lagged bundle where reads are known to have happened since — that means preservation did not engage, so check you used ai:reseed rather than ai:restore.

Adding a flag later: a pass-through flag reaches both entrypoints automatically (argument order is irrelevant). A flag that interacts with a pinned value must be added to that operation's pins in restore.mjs, or the operation will refuse it as a contradiction.

Pre-flight validation

Before any write touches a service, the orchestrator validates the bundle: the required subdirectories exist, every .jsonl in each declared bundle member is parseable (a torn write / corruption fails fast), the incident ledgers' non-JSONL heal-attempts.json and nested recovery-runs/*.jsonl are parsed too, and bundle-meta.json (when present) parses and passes the topology check. A torn or partial bundle aborts with a clear error and zero side effects on the live substrate.

Scope of that promise, stated precisely: validation covers files reachable from the declared bundle layout. It is not a guarantee about arbitrary content an operator drops into a bundle directory, and the restorability probe (verifyLatestBackupRestorable) reports RESTORABLE only when the bundle is both structurally valid and carries a non-zero row count — a bundle that parses cleanly while containing nothing is not a recovery source.

Production-target safeguard

When --mode replace targets canonical .neo-ai-data/ paths, the destructive-operation guard requires both an environment opt-in AND an explicit confirmation token before the truncate fires; otherwise the restore aborts. Disposable targets (under tmp/, the OS temp dir, or :memory: SQLite) bypass the requirement so tests can exercise replace mode safely.

Programmatic use

The orchestrator is also exported as runRestore({ bundleRoot, mode, force, forceTopologyMismatch, preserveReadState }) from ai/scripts/maintenance/restore.mjs, for embedding in higher-level recovery substrate (e.g. the daily snapshot pipeline or cold-restore harnesses). It returns per-subsystem result blocks, the parsed bundle-meta.json (or null for legacy bundles), and the topology-check verdict. Companion exports validateBundle(...) and checkTopology(...) expose the pre-flight checks for callers that want to gate before restoring.

Before you restore: is this bundle actually usable?

A backup that completed is not necessarily a backup you can restore from. A bundle can finish cleanly, write a receipt reading "status": "success", and contain zero rows — that status reports that the local bundle completed, which is a different fact from whether it holds your data.

npm run ai:check-backup-integrity

This is the artifact-verified timeline: it classifies every retained bundle by comparing each subsystem's manifest claim against the actual exported artifact bytes, so a manifest claiming 31,173 memories beside a zero-byte artifact is reported as manifest-false-green rather than trusted. Pass --backups /path/to/backups to point it at a non-default root, or --json for machine output.

Run it when any of these hold:

• Before any restore. The bundle you are about to restore from is the one claim you cannot afford to take on trust.
• The health surface reports lastSuccessful: null while count is non-zero. That means backups are running and none of them is usable — the opposite diagnosis from "backups are not running", and the opposite remedy.
• unusableCount in the health block is non-zero. It names how many completed bundles were disqualified; this tool names which ones and why.
• After any change to storage placement, volumes, or the persist path. A backup reads from wherever the topology says the data lives; if that moved and the backup did not, exports go quiet rather than loud.

Reading the output: clean is artifact-verified and restorable. manifest-false-green claims rows it does not have — do not restore from it. export-failed is the fail-loud path working correctly. no-mc-claim / empty-claim mean the subsystem exported nothing at all.

Reading bundle-meta.json: two blocks, two different questions

A bundle records two verdicts about a zero-row export, and they answer different questions. Read the right one for what you are deciding.

block	question	values	use it to decide
integrity[]	Survivability — does this bundle hold rows a restore could bring back?	pass · empty · fail · skipped	Whether to restore from this bundle.
capture.sources{}	Provenance — was there genuinely nothing to capture?	provenEmpty: true/false plus the facts behind it	Whether to investigate.

The two use different words on purpose. integrity[].status: "empty" says this bundle holds no rows for this subsystem; capture.sources[].provenEmpty says the facts establish there was nothing to capture. They are different claims about the same zero, and they can legitimately disagree.

Only integrity gates a restore. Any subsystem reading empty disqualifies the whole bundle — a partial restore is not a restore, and recovering memories while the knowledge base comes back with nothing is the kind of incompleteness that is hardest to notice afterwards. This holds no matter what the capture block says.

capture tells you whether a zero is expected. Each source records its row state, whether its collection identity is the same one the previous bundle saw (lineage), and derives provenEmpty only when both line up: a measured zero over an unchanged source.

• provenEmpty: true — a real empty source. Normal for a fresh environment.
• provenEmpty: false with lineage: "changed" — the one to investigate. The collection was replaced between captures. A re-embed or a restore does that deliberately with nothing lost, and so does a loss; the receipt refuses to guess. Correlate with maintenance history before concluding.
• provenEmpty: false with lineage: "unknown" — nothing to compare against. The first bundle after this field shipped reads this way, as does one whose predecessor was swept by retention.
• rowState: "unestablished" — the exporter returned something that was not a row count. The zero is a broken instrument, not a measurement; nothing about the corpus follows from it.

Legacy bundles degrade, they do not fail. A bundle published before the capture block existed carries no capture at all; it is still read correctly and still disqualifies exactly as it did. A bundle with no integrity block either resolves to restorable: null — unknown, which is a third answer and not a quiet "unusable"; check it with ai:check-backup-integrity above rather than assuming either way.

integrity[].status values never change spelling. A bundle is read by whatever version is deployed where it lands, and those readers match the status by exact string — so a renamed token reads to them as no empty subsystem found, i.e. a zero-row bundle reported as restorable. New meaning goes on new fields, never on this one.

Restoration Procedures

1. Knowledge Base (KB)

The Knowledge Base is the neo-knowledge-base collection inside the one flat unified ChromaDB store (ADR 0017) — a cache, not a store. Recover it by deterministic rebuild from source, at collection scope. Never delete the store folder: chroma/unified also holds the irreplaceable Memory Core collections.

Procedure:

1. Re-synchronize the KB from the source files (deterministic rebuild — clears and repopulates only the neo-knowledge-base collection):

   npm run ai:sync-kb
   

   (Note: Direct JSONL import of the kb/ bundle is deferred to #10871. Do not rm -rf the chroma/unified folder — it is shared with Memory Core; KB recovery is collection-scoped, handled by the rebuild above.)

2. Memory Core (MC) - Memories & Summaries

Memory Core memories and session summaries live as the neo-agent-memory and neo-agent-sessions collections inside the same flat unified ChromaDB store (ADR 0017). Unlike the KB, MC is the irreplaceable store — recover it from the backup bundle, at collection scope via the SDK. The pre-unification chroma/memory-core/ folder is retired.

Procedure:

1. Re-import the MC JSONL from the backup bundle via the SDK (mode: 'replace' clears and repopulates the MC collections at collection scope — no folder deletion):

   node -e "import('./ai/services.mjs').then(s => s.default.memory.manageDatabaseBackup({action: 'import', file: process.env.BUNDLE_ROOT + '/backup-<timestamp>/mc/memory-backup-<timestamp>.jsonl', mode: 'replace'}))"
   

   (Note: For full-bundle restores, prefer the Atomic-Bundle Restore CLI above. The direct SDK import remains the manual per-subsystem fallback. Do not rm -rf the chroma/unified folder — it is shared with the Knowledge Base; MC restore is collection-scoped via the SDK above.)

3. Chroma FTS5 Integrity Repair

The unified Chroma store is a shared physical SQLite database. pragma quick_check or pragma integrity_check can report malformed inverted index for FTS5 table main.embedding_fulltext_search while vector collections still answer normal queries. Treat this as a shared-store integrity incident: diagnose copy-first, stop all writers before touching the live database, and do not use Chroma defrag as a substitute for SQLite FTS5 repair.

Procedure:

1. Run the on-demand diagnostic and keep its copied SQLite snapshot:

   npm run ai:check-chroma-integrity -- --json --keep-snapshot
   

2. Validate the repair on the reported snapshot path, not on the live file:

   sqlite3 <snapshot>/chroma.sqlite3 "insert into embedding_fulltext_search(embedding_fulltext_search) values('rebuild'); pragma quick_check; pragma integrity_check;"
   

   Continue only if both pragmas return ok. If the copied snapshot remains malformed, stop here and recover from backup or rebuild the affected collection rather than experimenting on the live store.
3. Stop every process that can reach the Chroma daemon or the unified Chroma directory: Orchestrator, Memory Core, Knowledge Base, wake daemons, harness MCP server instances, and the Chroma daemon itself.
4. Capture a fresh backup bundle and a physical copy of the unified Chroma directory:

   npm run ai:backup
   cp -R .neo-ai-data/chroma/unified .neo-ai-data/chroma/unified.pre-fts5-rebuild-<timestamp>
   

5. Rebuild the live FTS5 table only after the writers are stopped and the backups exist:

   sqlite3 .neo-ai-data/chroma/unified/chroma.sqlite3 "insert into embedding_fulltext_search(embedding_fulltext_search) values('rebuild'); pragma quick_check; pragma integrity_check;"
   

6. Restart Chroma and the dependent AI services, then verify both SQLite integrity and API-level reachability:

   npm run ai:check-chroma-integrity -- --json
   node ai/scripts/maintenance/probeCollectionQueryHealth.mjs
   

Boundaries:

• ai/scripts/maintenance/defragChromaDB.mjs compacts collection storage; it is not an FTS5 integrity repair tool.
• KB rebuild (npm run ai:sync-kb) repairs the cache collection, not the shared SQLite full-text index.
• MC backup import restores collection rows, but it is not required when the copied FTS5 rebuild validates cleanly.
• API embedding-export failures such as Error finding id are a separate Chroma read-path issue (see §7 for its repair); do not conflate them with FTS5 index repair.

4. Memory Core - Native Edge Graph

The Memory Core Edge Graph is persisted in SQLite.

Procedure:

1. Move the corrupted SQLite database aside:

   mv .neo-ai-data/sqlite/memory-core-graph.sqlite .neo-ai-data/sqlite/memory-core-graph.sqlite.bak
   

2. Re-import the Graph JSONL from the backup bundle via the SDK:

   node -e "import('./ai/services.mjs').then(s => s.default.memory.manageDatabaseBackup({action: 'import', file: process.env.BUNDLE_ROOT + '/backup-<timestamp>/graph/graph-backup-<timestamp>.jsonl', mode: 'replace'}))"
   

5. Concept Ontology

The Concept Ontology consists of nodes and edges defined in JSONL.

Procedure:

1. Clear the active concepts directory:

   rm -rf .neo-ai-data/concepts/*
   

2. Copy the concepts from the backup bundle:

   cp -r "$BUNDLE_ROOT"/backup-<timestamp>/concepts/* .neo-ai-data/concepts/
   

6. RLAIF Trajectories

The RLAIF trajectories capture interaction feedback and metadata for offline RL alignment.

Procedure:

1. Replace the active trajectories file:

   cp "$BUNDLE_ROOT"/backup-<timestamp>/trajectories/trajectories.jsonl .neo-ai-data/datasets/rlaif/trajectories.jsonl
   

7. Memory Core Stored-Embedding Export Repair

A distinct failure from §3 (FTS5) and from a full restore (§2): npm run ai:check-chroma-integrity reports get embedding by id: Error finding id (stored-embedding export fails) for neo-agent-memory / neo-agent-sessions / neo-native-graph while the query canary stays healthy — Chroma metadata rows are present but many ids are missing from the persisted HNSW vector index (#13496 / #13467). Backup/export silently skips the missing-vector ids, so it is not a safe substitute. The repair re-embeds the missing vectors into a shadow collection and promotes copy-first (defragChromaDB.mjs repairMemoryCoreCollectionsViaFullEnumeration, #13635), behind the --allow-memory-core opt-in (default fails closed).

Procedure:

1. Confirm the diagnosis (read-only — a healthy query canary plus a failing exportability sample is the signature):

   node ai/scripts/maintenance/probeCollectionQueryHealth.mjs
   npm run ai:check-chroma-integrity -- --exportability-sample-size 2 --json
   

2. Quiesce every competing writer — but leave the Chroma server running (both the backup and the repair use its API; unlike §3's file-level FTS5 repair, this does not stop Chroma). Stop the Orchestrator, Memory Core, Knowledge Base, wake daemons, and harness MCP server instances (the npm run ai:server processes) so nothing else mutates the collections during the repair.
3. With the writers stopped (store quiescent, Chroma still up), capture the canonical SDK backup and a coarse physical rollback copy:

   npm run ai:backup
   cp -R .neo-ai-data/chroma/unified .neo-ai-data/chroma/unified.pre-mc-repair-<timestamp>
   

   (ai:backup reads through Chroma's API, so Chroma must be up; the physical copy is a coarse rollback taken while no writers are active. defragChromaDB additionally takes its own private pre-promote snapshot.)
4. Run the repair with Chroma running and no competing writers (the repair is the exclusive collection writer; opt-in — it shadow-extracts intact vectors, re-embeds the missing ids, validates the shadow collection, then promotes copy-first):

   node ai/scripts/maintenance/defragChromaDB.mjs --target memory-core --allow-memory-core
   

   A clean run clears its repair state-marker; a partial run rewrites an explicit memory-core-repair-aborted marker and exits non-zero — investigate before re-running.
5. Verify export and query health are both green, then restart the AI services (see Verification below):

   npm run ai:check-chroma-integrity -- --json
   node ai/scripts/maintenance/probeCollectionQueryHealth.mjs
   

Boundaries:

• This repairs stored-embedding export (re-embed missing vectors); it is not the §3 FTS5 SQLite repair and not the §2 backup-restore — use those for their respective failures.
• The repair is gated behind --allow-memory-core (default fails closed) and promotes copy-first; it never deletes/recreates the live collection in place.
• The heavier exportability probe (get embedding by id) stays on-demand (the maintenance scripts above); the routine bounded healthcheck remains query-path only.

Verification

After restoration, restart the subsystem and verify health.

npm run ai:server

Validate recovery by ensuring healthcheck.backup.lastSuccessful surfaces your desired timestamp and no subsystem connection errors are reported.

Important: Never perform destructive restores on shared multi-tenant databases without confirming coordinate isolation. Only restore into intended environments.