Migration Guide

June 22, 2026 · View on GitHub

This document provides step-by-step instructions for migrating between different AutoMem configurations.

Heads up for existing deployments: The default embedding dimension is now 1024d (voyage-4) for new installs. If your Qdrant collection uses a different dimension (e.g. 3072d from text-embedding-3-large or 768d from text-embedding-3-small), no action is needed — VECTOR_SIZE_AUTODETECT=true (the default) automatically adopts your existing collection dimension on startup. To explicitly pin your dimension, set VECTOR_SIZE=<your-dimension> in your .env. To enforce strict matching (fail on mismatch), set VECTOR_SIZE_AUTODETECT=false.

Table of Contents

Embedding dimension migrations

Data & schema migrations

Reference

Every script below is cataloged in scripts/README.md. The embedding-dimension sections are about re-vectorizing; the data & schema sections are one-time, idempotent FalkorDB/Qdrant migrations.

Migrating to 1024d (voyage-4 default)

When to migrate: If you're switching from OpenAI embeddings to Voyage AI (the new recommended default).

Steps

  1. Backup your data: python scripts/backup_automem.py
  2. Set environment variables: 
    EMBEDDING_PROVIDER=voyage    # or auto (will prefer Voyage if VOYAGE_API_KEY is set)
VOYAGE_API_KEY=pa-...
VECTOR_SIZE=1024
  3. Delete and recreate the Qdrant collection: 
    curl -X DELETE http://localhost:6333/collections/memories
  4. Re-embed all memories: 
    python scripts/reembed_embeddings.py
  5. Verify: Check that /health shows vector_size: 1024 and recall returns results.

Alternatively, set VECTOR_SIZE_AUTODETECT=true (the default) and AutoMem will adopt your existing collection dimension without migration. Only migrate when you want to switch embedding providers.

Upgrading to 3072d Embeddings

When to upgrade: If you need better semantic precision and have the storage budget for 4x larger embeddings.

Pros ✅

  • Better semantic precision: ~5-10% improvement on benchmarks
  • Improved multi-hop reasoning: Better at connecting related concepts
  • Recommended for production: If accuracy is critical and storage is not a constraint

Cons ❌

  • 4x storage cost: 768 → 3072 dimensions (4x more disk space)
  • 4x embedding cost: OpenAI charges per dimension
  • ~20% slower search: More dimensions = more computation
  • Migration required: Cannot reuse existing embeddings

Cost Comparison

Metric768d (small)3072d (large)Multiplier
Storage per 1M memories~3GB~12GB4x
OpenAI cost per 1M tokens$0.02$0.136.5x
Search latency~50ms~60ms1.2x
Benchmark accuracy88.2%90.5%+2.3pp

Migration Steps

1. Backup Your Data

python scripts/backup_automem.py

This creates timestamped backups in backups/:

  • backups/falkordb/memories_YYYYMMDD_HHMMSS.rdb
  • backups/qdrant/qdrant_snapshot_YYYYMMDD_HHMMSS.tar.gz

2. Update Configuration

# Add to your .env file
echo "VECTOR_SIZE=3072" >> .env
echo "EMBEDDING_MODEL=text-embedding-3-large" >> .env

Or export temporarily:

export VECTOR_SIZE=3072
export EMBEDDING_MODEL=text-embedding-3-large

3. Re-embed All Memories

python scripts/reembed_embeddings.py

This will:

  • Fetch all memories from FalkorDB (source of truth)
  • Generate new 3072d embeddings using OpenAI API
  • Recreate Qdrant collection with new dimensions
  • Upsert all embeddings in batches

Expected time: ~5-10 minutes per 10k memories

4. Verify Migration

Check Qdrant collection info:

curl http://localhost:6333/collections/memories | jq '.result.config.params.vectors'

Should show:

{
  "size": 3072,
  "distance": "Cosine"
}

5. Test Recall

curl -X POST http://localhost:8001/recall \
  -H "Authorization: Bearer $AUTOMEM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "test recall", "limit": 5}'

Verify results are returned and scores look reasonable.

6. Restart Application

# If using Docker
docker compose up -d

# Or rerun the foreground dev stack
make dev

# If using systemd
sudo systemctl restart automem

# If using Railway
railway up

Rollback Procedure

If migration fails or results are poor:

# 1. Stop application
docker compose down  # or however you run AutoMem

# 2. Restore from backup
python scripts/restore_from_backup.py backups/qdrant/qdrant_snapshot_YYYYMMDD_HHMMSS.tar.gz
python scripts/restore_from_backup.py backups/falkordb/memories_YYYYMMDD_HHMMSS.rdb

# 3. Revert configuration
export VECTOR_SIZE=768
export EMBEDDING_MODEL=text-embedding-3-small

# 4. Restart
docker compose up -d

Downgrading to 768d Embeddings

When to downgrade: If storage costs are too high or 3072d isn't providing enough value.

Steps

Follow the same migration steps above, but use:

export VECTOR_SIZE=768
export EMBEDDING_MODEL=text-embedding-3-small

Then run reembed_embeddings.py to recreate the collection with 768d vectors.

Upgrading to 0.16.0

0.16.0 adds several one-time, idempotent data migrations. None are required to keep an existing instance running — run them to adopt the new behavior. Each is safe to re-run and most support --dry-run. Back up first (python scripts/backup_automem.py).

Promote entity tags to first-class Entity nodes

Earlier versions recorded entities only as entity:{category}:{slug} tags on Memory nodes. This migration creates real Entity nodes and links them with REFERENCED_IN relationships, which enables entity-centric recall and graph queries.

python scripts/migrate_entity_nodes.py --dry-run   # preview
python scripts/migrate_entity_nodes.py             # apply (idempotent)

First-class entity-node synthesis (IDENTITY_SYNTHESIS_ENABLED) is experimental and gated off by default while people-entity word-pair noise is addressed (#181). Promote nodes when you opt into that feature. If your clone has noisy generated entity tags, clean them first with scripts/lab/repair_entity_tags.py (--mode audit → review the plan → --mode execute, with --mode rollback available).

Backfill tag prefixes

Recomputes the tag_prefixes sidecar on every memory in FalkorDB and Qdrant so prefix-match recall stays consistent with the current tag set. Run once after upgrading.

python scripts/backfill_tag_prefixes.py

Rescore relevance

Recomputes every relevance_score with the corrected consolidation decay (base_decay_rate=0.01 + importance floor), undoing the damage from the old over-aggressive 0.1 rate. Targets a local clone by default; pass --target to point at production.

python scripts/rescore_relevance.py --dry-run     # preview against local
python scripts/rescore_relevance.py               # apply against local

Clean up invalid memory types

Reclassifies legacy/invalid type values (e.g. session_start, interaction) back to the valid set (Decision, Pattern, Preference, Style, Habit, Insight, Context). Reads .env; takes no flags.

python scripts/cleanup_memory_types.py

Importing from the legacy MCP SQLite store

If you're moving off the old MCP memory service, replay its sqlite_vec.db into AutoMem via the API. Original timestamps, tags, and importance are preserved, and the legacy payload is kept under metadata['legacy']. Always --dry-run first.

# Preview what will be imported
python scripts/migrate_mcp_sqlite.py --dry-run

# Run the import against a deployed instance
python scripts/migrate_mcp_sqlite.py \
  --db /path/to/sqlite_vec.db \
  --automem-url https://automem.example.com \
  --api-token $AUTOMEM_API_TOKEN

# Refresh embeddings afterward
python scripts/reembed_embeddings.py --limit 200

Troubleshooting

Error: "Vector dimension mismatch"

Symptom (only when VECTOR_SIZE_AUTODETECT=false):

FATAL: Vector dimension mismatch detected!
  Existing Qdrant collection: 3072d
  Configured VECTOR_SIZE:     1024d

Solution (pick one):

  1. Set VECTOR_SIZE_AUTODETECT=true (default) to automatically adopt the existing collection dimension
  2. Set VECTOR_SIZE=<existing-dimension> in your .env to match your data
  3. Migrate to the new dimension: follow the 1024d, 3072d, or 768d migration steps above

Error: "OpenAI API rate limit"

Symptom:

Rate limit exceeded during re-embedding

Solution: The reembed_embeddings.py script uses whatever embedding provider is configured (Voyage, OpenAI, local, etc.). For large datasets:

  1. Run during off-peak hours
  2. Increase your provider's rate limits if applicable
  3. Split migration into batches using --batch-size flag

Error: "Qdrant collection already exists"

Symptom:

Collection 'memories' already exists with different dimension

Solution: Delete and recreate:

curl -X DELETE http://localhost:6333/collections/memories
python scripts/reembed_embeddings.py

⚠️ Warning: This deletes all embeddings. Make sure FalkorDB still has the memories (embeddings will be regenerated from there).

Migration is slow

Symptoms:

  • Taking hours for thousands of memories
  • High OpenAI API costs

Solutions:

  1. Check batch size: Script uses batches of 100 by default
  2. Parallel processing: Use --workers flag (if implemented)
  3. Spot check first: Test on a subset before full migration
  4. Use cheaper model for testing: 
    export EMBEDDING_MODEL=text-embedding-3-small
python scripts/reembed_embeddings.py --dry-run

Backup failed

Symptoms:

  • Backup script errors
  • Empty backup files

Solutions:

  1. Check disk space: df -h
  2. Check permissions: ls -la backups/
  3. Manual backup: 
    # FalkorDB
docker exec automem-falkordb-1 redis-cli --rdb /data/dump.rdb

# Qdrant
curl -X POST http://localhost:6333/collections/memories/snapshots

Best Practices

Before Any Migration

  1. ✅ Always backup first - Don't skip this step
  2. ✅ Test in staging - If you have a staging environment
  3. ✅ Monitor costs - Check OpenAI usage dashboard during migration
  4. ✅ Document current state - Note current VECTOR_SIZE and EMBEDDING_MODEL

After Migration

  1. ✅ Run benchmark tests - Verify accuracy hasn't degraded
  2. ✅ Monitor performance - Check search latency and throughput
  3. ✅ Update documentation - Note when migration occurred and why
  4. ✅ Store migration record: 
    curl -X POST http://localhost:8001/memory \
  -H "Authorization: Bearer $AUTOMEM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Migrated to 3072d embeddings for better semantic precision",
    "tags": ["migration", "config", "embeddings"],
    "importance": 0.8
  }'

Choosing Between 768d and 3072d

Use 768d (text-embedding-3-small) if:

  • Cost-conscious deployment
  • Storage is limited
  • Speed > slight accuracy gains
  • Personal/development use
  • Small dataset (<100k memories)

Use 3072d (text-embedding-3-large) if:

  • Production deployment
  • Accuracy is critical
  • Complex multi-hop reasoning needed
  • Large dataset benefits from precision
  • Storage/compute costs are acceptable