Index Deployment & Troubleshooting Guide

June 9, 2026 · View on GitHub

Purpose: Provide a single, opinionated, end-to-end reference for standing up, promoting, operating, and troubleshooting Index instances across development, staging, and production while preserving governance guarantees and observability.

1. Deployment Profiles

ProfileGoalCharacteristicsTypical Settings
Dev (workbench)Rapid iteration & feature validationLocal filesystem, verbose logs, mutation enabledINDEX_SERVER_VERBOSE_LOGGING=1, INDEX_SERVER_DEBUG=1, INDEX_SERVER_DASHBOARD=1
Shared Dev / IntegrationCross-developer validation & test harnessStable path, persistent metrics, controlled mutationINDEX_SERVER_METRICS_FILE_STORAGE=1, optional INDEX_SERVER_BOOTSTRAP_AUTOCONFIRM=1 (tests)
Staging / Pre-ProdRelease candidate soakMirrors prod paths & policies, seeded baselineSame as prod + extra diagnostics INDEX_SERVER_LOG_DIAG=1 if needed
ProductionStable Index serving & governanceLocked paths, minimal verbosity, audit loggingDefault writes + bootstrap gating, no verbose/diag unless incident
Reference / Read-OnlyImmutable published snapshotAll mutation permanently disabledINDEX_SERVER_REFERENCE_MODE=1
<mcp-root>/
  index-server/
    instructions/           # Auto-created by server on first startup (canonical seeds)
    dist/                   # Built server artifacts
    logs/ (optional)        # Central log aggregation target (if not using default)
    metrics/ (optional)     # If using file-backed metrics ring
  index-server-stage/   # Staging mirror
  index-server-archive/ # Historical snapshots (optional)

Keep dev working copy separate: e.g. <root>/index-server pointing INDEX_SERVER_DIR to an external folder to avoid accidental mutation of production index during debugging. The repo root does not contain an instructions/ directory — it is only created at the deployment target.

3. Bootstrap & Auto-Seeding

On startup the server guarantees presence of two baseline governance instructions:

  • 000-bootstrapper
  • 001-lifecycle-bootstrap

They are auto-created (idempotent, non-destructive) if missing unless you set:

INDEX_SERVER_AUTO_SEED=0

Verbose seeding diagnostics:

INDEX_SERVER_SEED_VERBOSE=1

Structured log event: seed_summary with fields: created, existing, disabled, hash (deterministic canonical hash for auditing).

When Copying Production Instructions to Dev

If you copy instructions/ from production into a dev directory:

  • The seeding system detects both seeds already exist and emits seed_summary with created=[], existing=[...].
  • No overwrites occur.
  • If you only copy a subset and omit a seed, the missing one will be recreated—this is safe and expected.

Bootstrap Confirmation Flow Recap

  1. Fresh workspace (only seeds) → mutation gated: bootstrap_confirmation_required.
  2. Call bootstrap with action=request (or legacy bootstrap_request) → get token.
  3. Human approves; call bootstrap with action=confirm (or legacy admin bootstrap_confirmFinalize) submitting the token.
  4. Confirmation artifact bootstrap.confirmed.json persists.
  5. Any additional instruction beyond the seeds implicitly means existing workspace (confirmation optional).
  6. INDEX_SERVER_REFERENCE_MODE=1 short-circuits everything: Index immutable forever.

Test harness shortcut: INDEX_SERVER_BOOTSTRAP_AUTOCONFIRM=1 (never use in prod).

4. Environment Variable Matrix (Key Operational Controls)

VariablePurposeTypical ProdTypical DevMigration Notes
INDEX_SERVER_DIRIndex rootStable prod pathdevinstructions/(Will be normalized to INDEX_SERVER_DIR alias internally)
INDEX_SERVER_MUTATIONForce read-only overrideunset / 1unset / 1Set 0 when a deployment must be explicitly read-only
INDEX_SERVER_MESSAGING_ENABLEDEnable inter-agent messaging subsystem11Set 0 to remove all messaging_* MCP tools, skip the dashboard REST routes, and hide the Messaging tab. See #353.
INDEX_SERVER_REFERENCE_MODEForce read-only00 or 1 (testing)Unchanged
INDEX_SERVER_AUTO_SEEDAuto-create baseline seeds11Unchanged
INDEX_SERVER_SEED_VERBOSEExtra stderr seed log01Unchanged
INDEX_SERVER_LOG_LEVELUnified log levelinfodebugConsolidates verbose/diag flags over time
INDEX_SERVER_VERBOSE_LOGGINGLegacy verbose toggle01Deprecated (maps to INDEX_SERVER_LOG_LEVEL=debug)
INDEX_SERVER_LOG_DIAGLegacy diagnostic toggle01Deprecated (maps to INDEX_SERVER_LOG_LEVEL=trace)
INDEX_SERVER_TRACEFine-grained trace tokensmanifest selectivelymanifest,bootstrapUse tokens instead of new booleans
INDEX_SERVER_TRACE_FILEStructured tracing file01 (targeted)Unchanged
INDEX_SERVER_METRICS_FILE_STORAGEPersist metrics ring11 or 0Unchanged
INDEX_SERVER_METRICS_MAX_FILESMetrics rotation depth720120Unchanged
INDEX_SERVER_BOOTSTRAP_AUTOCONFIRMTest auto-confirm01Test only
INDEX_SERVER_LOG_SYNCSynchronous log fsync (tests)01Test determinism only
INDEX_SERVER_TIMING_JSONStructured timing overridesMinimalRich (tests)Replaces ad-hoc MANIFEST_TEST_WAIT_* vars
INDEX_SERVER_TEST_MODETest/coverage mode(unset)coverage-fastReplaces FAST_COVERAGE=1
COVERAGE_HARD_MINCoverage gate (hard fail)e.g. 50e.g. 50Accessed via runtime config
COVERAGE_TARGETAdvisory coverage targete.g. 60e.g. 60Accessed via runtime config

Consolidation Note: Introduce no new top-level environment variables without first attempting to express the need via INDEX_SERVER_TIMING_JSON, INDEX_SERVER_TRACE, or an extension of runtimeConfig. File src/config/runtimeConfig.ts is the single source of truth for mapping and deprecation warnings.

Example structured timing override (PowerShell):

$env:INDEX_SERVER_TIMING_JSON = '{"manifest.waitDisabled":18000,"manifest.waitRepair":20000,"manifest.postKill":300}'

Then in tests/services:

import { getRuntimeConfig } from '../src/config/runtimeConfig';
const cfg = getRuntimeConfig();
const waitRepair = cfg.timing('manifest.waitRepair', 20000);

Legacy Timing Flags (still honored with one-time warnings):

  • MANIFEST_TEST_WAIT_DISABLED_MS → INDEX_SERVER_TIMING_JSON: manifest.waitDisabled
  • MANIFEST_TEST_WAIT_REPAIR_MS → INDEX_SERVER_TIMING_JSON: manifest.waitRepair

Fast coverage path migration:

  • Old: FAST_COVERAGE=1
  • New: INDEX_SERVER_TEST_MODE=coverage-fast

Mutation control migration:

  • Old habit: INDEX_SERVER_MUTATION=1
  • New default: leave INDEX_SERVER_MUTATION unset for normal writes
  • Read-only override: INDEX_SERVER_MUTATION=0

5. Deployment Workflow

5.1 Build Artifact

npm ci
npm run build

Artifacts: dist/server/index-server.js plus dashboard assets (copied by scripts/copy-dashboard-assets.mjs).

5.2 Promote to Target

Use provided PowerShell script:

pwsh -File scripts/deploy-local.ps1 -Rebuild -Overwrite -TargetDir <production-install-root>

Flags:

  • -Rebuild – runs npm ci && npm run build prior to copy
  • -Overwrite – replaces existing target directory
  • Production dependencies are always installed using npm ci --production (lock file ensures deterministic versions)

To enable embedding-based semantic search, set the environment variable before starting:

export INDEX_SERVER_SEMANTIC_ENABLED=1

$env:INDEX_SERVER_SEMANTIC_ENABLED = '1'

The first search request downloads a ~90MB embedding model (one-time). Subsequent requests use the cached model.

Vector Embedding Storage (sqlite-vec)

When using the SQLite storage backend, embeddings are stored in a sqlite-vec vec0 virtual table for native KNN search automatically. The INDEX_SERVER_SQLITE_VEC_ENABLED flag is auto-enabled when backend=sqlite — explicit =1 is optional, and =0 opts out:

export INDEX_SERVER_STORAGE_BACKEND=sqlite
export INDEX_SERVER_SEMANTIC_ENABLED=1
# INDEX_SERVER_SQLITE_VEC_ENABLED defaults to on; set =0 to opt out

$env:INDEX_SERVER_STORAGE_BACKEND = 'sqlite'
$env:INDEX_SERVER_SEMANTIC_ENABLED = '1'
# INDEX_SERVER_SQLITE_VEC_ENABLED defaults to on; set '0' to opt out

Requirements:

  • Node.js ≥ 22.13.0 (for DatabaseSync.loadExtension())
  • sqlite-vec npm package (bundled in production dependencies)

If sqlite-vec fails to load (e.g., missing native binary on Alpine Linux), embeddings fall back to JSON storage automatically.

Node.js Version Matrix

FeatureMinimum Node.jsReason
JSON backend≥ 20Standard Node.js APIs
SQLite backend≥ 22.5.0Built-in node:sqlite module
sqlite-vec embeddings≥ 22.13.0DatabaseSync.loadExtension() API

5.3 First Start (Prod)

# Example (stdio integration client config points cwd here):
node dist/server/index-server.js --dashboard-port=8787

Verify logs (stderr) contain:

  • [startup] Dashboard server started successfully (if dashboard enabled)
  • seed_summary (first start or hash check) – confirm existing vs created
  • server_started

5.4 Validation Checklist

ItemCommand / ToolExpectation
Seed Summarylogs / seed_summarycreated=2 (fresh) or created=0 (existing)
Mutation Gatetools/call bootstrap_statusrequireConfirmation=false (existing) or true (fresh)
Instructions Healthtools/call index_healthrecursionRisk=none
Metrics Snapshottools/call metrics_snapshottool counts increment on calls

6. Copying Production Instructions to Dev (Troubleshooting Scenario)

You mentioned copying production instructions into dev to reproduce an issue. Recommended steps:

  1. Decide isolation path: create devinstructions-prod-clone/.
  2. Copy: robocopy <production-install-root>\instructions <root>\index-server\devinstructions /E
  3. Point dev config (.vscode/mcp.json) INDEX_SERVER_DIR to cloned folder.
  4. Start server with verbose flags: INDEX_SERVER_VERBOSE_LOGGING=1 INDEX_SERVER_LOG_DIAG=1.
  5. Run targeted test or reproduce workflow.
  6. Compare logs vs prod baseline. Key events: Index-summary, tool_start/tool_end, seed_summary, bootstrap_status.
  7. After debugging, discard clone to avoid accidental mutation of real prod snapshot.

If you only copied some files and lost a seed, auto-seed reintroduces it—this is safe. To detect divergence, compare the hash in seed_summary between environments; mismatch after manual edits signals drift.

7. Troubleshooting Matrix

SymptomLikely CauseAction
All logs show [warning] [server stderr]MCP log bridge not activatingVerify mcpLogBridge is the first import in index-server.ts. Check handshake completion: look for activateMcpLogBridge() call in logs. See mcp_stdio_logging.md.
No log output in VS Code Output panelOutput channel not selected or server not startedOpen View → Output, select the index-server channel from the dropdown. Verify server is running (check status bar).
Logs show wrong severity levelsCustom inferLevel or NDJSON format mismatchCheck that log lines are valid NDJSON with a "level" field. The defaultInferLevel() function reads the "level" field first, then falls back to keyword matching.
INDEX_SERVER_LOG_FILE not producing outputPath doesn't exist or no write permissionSet INDEX_SERVER_LOG_FILE=1 for default path (logs/mcp-server.log), or set to absolute path. Ensure logs/ directory exists or is auto-created.
Startup logs missing (only post-handshake visible)Buffer overflow or early crashIncrease maxBufferSize (default 500 lines). Check for early exceptions in stderr before bridge activates.
Mutation blocked unexpectedlyMissing confirmation or reference modeCall bootstrap_status; if requireConfirmation=true, complete token flow. Check INDEX_SERVER_REFERENCE_MODE.
Seeds recreated on existing workspaceSeeds deleted manuallyAccept recreation; investigate deletion; enable INDEX_SERVER_SEED_VERBOSE=1 for audit timing.
No seed_summary lineLogging misconfigured or very early crashEnsure INDEX_SERVER_LOG_FILE=1; confirm autoSeedBootstrap() runs before Index usage; inspect stderr for stack traces.
Tool calls lack tool_endAsynchronous logging flush race in testsUse INDEX_SERVER_LOG_SYNC=1 (test only) or increase polling window.
Drift in governance hashManual edits without bumping versionRun governanceHash tests; re-export canonical spec; version increment.
Dashboard won’t startPort in use / blockedUse --dashboard-port=<free> or set INDEX_SERVER_DASHBOARD_PORT; check firewall.
Index shows zero instructionsWrong INDEX_SERVER_DIRConfirm path & permissions; check stderr [startup] toolsRegistered... instructionsDir="...".

8. Observability Signals

Key structured events (JSON logs):

  • logger_init – file log path, size
  • seed_summary – seeding outcome
  • Index-summary – counts (scanned / accepted / skipped) + salvage
  • tool_start / tool_end / tool_error – lifecycle timing + correlation
  • bootstrap_status (via tool) – current gating state

Aggregate or forward these into your logging system for RUM or audit trails. Correlate by timestamp or add a future correlation ID if centralization requires cross-instance stitching.

9. Hardening Recommendations

AreaControl
IntegrityPeriodic integrity job computes canonical seed hash & compares to seed_summary.hash.
BackupSnapshot instructions/ + metrics/ nightly.
PromotionGit-based PR review for instruction changes; promote via controlled import tool.
Drift DetectionScheduled tool invoking Index-summary & diffing against last baseline snapshot.
AccessFile ACL restrict write to service account; devs mutate via controlled workflow only.
Dashboard TLSFor local/internal/dev: index-server --init-cert --start --dashboard bootstraps a self-signed cert+key (see cert_init.md). For internet-exposed or production services use a CA-issued cert from your existing PKI — --init-cert is intentionally a developer/internal-loopback convenience and v1 does not auto-install into OS trust stores.

10. FAQ

Q: How do I fully reset a dev workspace?
A: Delete the dev instructions directory contents; restart server. Seeds auto-reappear; confirmation gating re-engages (unless non-seed files added).

Q: How do I simulate production read-only mode?
A: Set INDEX_SERVER_REFERENCE_MODE=1; seeds load but mutation tools return block reason reference_mode_read_only.

Q: Can I disable seeding for a forensic run?
A: Yes: INDEX_SERVER_AUTO_SEED=0; if seeds absent you may hit gating conditions; manually copy seeds if needed for consistent bootstrap path.

11. Future Enhancements (Planned / Optional)

  • Seed integrity enforcement: warn if on-disk seed differs from canonical JSON (without overwriting).
  • Signed Index manifests for tamper detection.
  • Distributed lock / notification for multi-node Index mutation coordination (post baseline).

12. Quick Reference Commands

# Build & deploy (local prod)
pwsh -File scripts\deploy-local.ps1 -Rebuild -Overwrite -TargetDir <production-install-root>

# Start dev with verbose logging
$env:INDEX_SERVER_DIR='<root>/index-server/devinstructions'; \
$env:INDEX_SERVER_VERBOSE_LOGGING='1'; $env:INDEX_SERVER_DASHBOARD='1'; node dist/server/index-server.js --dashboard-port=8787

# Check bootstrap status (example RPC via client tooling)
# tools/call name=bootstrap_status

# Metrics snapshot
# tools/call name=metrics_snapshot

13. Change Log (Document)

  • v1.0: Initial creation with auto-seeding & troubleshooting guidance (2025-09-15)

Happy deploying – this guide should give you everything needed to reproduce prod locally, ensure seeds are present, and safely iterate.