Testing Guide

July 7, 2026 · View on GitHub

This document describes the testing setup for AutoMem and how to run tests against different environments.

Test Suite Overview

AutoMem has a comprehensive test suite covering:

  • API endpoints
  • Core functionality
  • Consolidation engine
  • Enrichment pipeline
  • Integration tests

Quick Commands

# Unit tests only (fast, no services required)
make test

# Integration tests (local Docker)
make test-integration

# Integration tests (live Railway server)
make test-live

Pytest Markers

The suite now uses explicit markers:

  • unit: Fast mocked tests (default classification).
  • integration: Docker-backed local integration tests.
  • live: Benchmarks/live-environment tests (opt-in).

Marker behavior:

  • Any test not explicitly marked integration or live is auto-classified as unit.
  • make test runs -m unit.
  • make test-integration runs -m integration with the existing env-based gate.

Test Types

flowchart TD
    Start{What are you<br/>testing?}

    Start -->|Code changes| CodeCheck{Changed<br/>API/DB logic?}
    Start -->|Pre-commit| PreCommit[Run: make test<br/>✅ Unit tests only<br/>⚡ Fast 10s]
    Start -->|Before deploy| PreDeploy{Have Railway<br/>deployed?}
    Start -->|Benchmarking| Benchmark[Run: make test-locomo<br/>📊 LoCoMo benchmark<br/>⏱️ 15-25 min]

    CodeCheck -->|No just refactor| PreCommit
    CodeCheck -->|Yes API/DB changes| Integration{Need real<br/>services?}

    Integration -->|Yes| Docker{Have Docker<br/>running?}
    Integration -->|No mock is fine| PreCommit

    Docker -->|Yes| LocalIntegration[Run: make test-integration<br/>🐳 Local Docker stack<br/>⏱️ 60s]
    Docker -->|No| LiveCheck{Have Railway<br/>access?}

    PreDeploy -->|Yes| LiveIntegration[Run: make test-live<br/>☁️ Test against Railway<br/>⏱️ 90s]
    PreDeploy -->|No| LocalIntegration

    LiveCheck -->|Yes| LiveIntegration
    LiveCheck -->|No| SetupDocker[Set up Docker first<br/>docker compose up]

    SetupDocker --> LocalIntegration

    LocalIntegration --> Success{Tests pass?}
    LiveIntegration --> Success
    PreCommit --> Success
    Benchmark --> BenchSuccess{Score acceptable?}

    Success -->|Yes| Commit[✅ Safe to commit/deploy]
    Success -->|No| Debug[🔍 Debug failures<br/>Check logs]

    BenchSuccess -->|Yes| Commit
    BenchSuccess -->|No| Optimize[🔧 Optimize recall logic]

1. Unit Tests

Command: make test

  • Fast, isolated tests using mock/stub implementations
  • No external services required
  • Tests API logic, validation, edge cases
  • Safe to run anytime
  • Internally runs: pytest -m unit

2. Integration Tests (Local)

Command: make test-integration

  • Tests against real Docker services (FalkorDB + Qdrant + API)
  • Automatically starts services with test credentials
  • Creates test memories tagged with ["test", "integration"]
  • Cleans up all test data after completion
  • Requires: Docker, Docker Compose
  • Internally runs: pytest -m integration

What it does:

  1. Starts Docker services with AUTOMEM_API_TOKEN=test-token
  2. Waits for services to be ready (5s)
  3. Runs full integration test suite
  4. Tests real database operations, embeddings, associations

3. Live Server Tests (Railway)

Command: make test-live

  • Tests against the production Railway deployment
  • Verifies local and live environments have matching behavior
  • Prompts for confirmation before running (safety measure)
  • Automatically fetches Railway credentials
  • Requires: Railway CLI, linked project (railway link)

Safety features:

  • Interactive confirmation required
  • Only creates/modifies test memories with unique UUIDs
  • All test data is cleaned up immediately
  • Read-only operations for health checks and recalls

Test Scripts

Interactive Live Testing

./test-live-server.sh

Prompts for confirmation before running against production.

Automated Live Testing

./test-live-server.sh --non-interactive

Non-interactive flag for CI/automation. The legacy test-live-server-auto.sh wraps this flag for backward compatibility.

Manual Integration Testing

./run-integration-tests.sh

Runs integration tests with proper environment setup.

Environment Variables

Required for Integration Tests

  • AUTOMEM_RUN_INTEGRATION_TESTS=1 - enables integration tests
  • AUTOMEM_TEST_API_TOKEN - API authentication token
  • AUTOMEM_TEST_ADMIN_TOKEN - admin authentication token (optional for some tests)

Optional Configuration

  • AUTOMEM_TEST_BASE_URL - override API endpoint (default: http://localhost:8001)
  • AUTOMEM_ALLOW_LIVE=1 - required to test against non-localhost URLs
  • AUTOMEM_START_DOCKER=1 - auto-start Docker services
  • AUTOMEM_STOP_DOCKER=1 - auto-stop Docker after tests (default)

Test Results

All tests are designed to pass cleanly with no unexpected warnings (filtered via pytest.ini). Use make test, make test-integration, or make test-live and rely on the pytest summary for the current counts.

Comparing Local vs Live

To verify local Docker environment matches production:

# Run tests locally
make test-integration

# Run same tests against live
make test-live

Both should produce identical results, confirming:

  • API responses match
  • Authentication works correctly
  • Database operations behave the same
  • Embeddings are generated consistently

Troubleshooting

"API not available" error

The integration tests wait up to 10 seconds for the API to be ready. If services take longer:

  • Check docker compose ps to see service status
  • Check docker compose logs flask-api for startup errors
  • Manually verify health: curl http://localhost:8001/health

"Unauthorized" errors (401)

Ensure environment variables match:

  • Local: AUTOMEM_API_TOKEN=test-token
  • Docker: Set via docker-compose.yml environment section
  • Railway: Check with railway variables

Railway CLI issues

# Install Railway CLI
npm install -g @railway/cli

# Link to project
railway link

# Verify connection
railway status

CI/CD Integration

For automated testing in CI:

# Unit tests (always safe)
make test

# Integration tests (if Docker available)
make test-integration

# Live tests (if Railway credentials available)
./test-live-server.sh --non-interactive

LoCoMo Benchmark

AutoMem can be evaluated against the LoCoMo benchmark (ACL 2024), which tests long-term conversational memory across 10 conversations and 1,986 questions.

LoCoMo Cat-5 Judge

Category 5 uses evidence-grounded complex reasoning and is opt-in for cost reasons.

# Default: categories 1-4 scored, category 5 skipped
make bench-eval BENCH=locomo-mini CONFIG=baseline

# Enable cat-5 judge with env var
BENCH_JUDGE_MODEL=gpt-5.4-mini-2026-03-17 make bench-eval BENCH=locomo-mini CONFIG=baseline

# Direct runner: 1 conversation, judge enabled
./test-locomo-benchmark.sh --conversations 0 --judge --output benchmarks/results/locomo-smoke.json

# Direct runner: mini mode (same 2 conversations as locomo-mini)
./test-locomo-benchmark.sh --conversations 0,1 --judge --output benchmarks/results/locomo-mini.json

Relevant flags for ./test-locomo-benchmark.sh:

  • --conversations I,J,... runs only the selected 0-based conversation indices from locomo10.json. Default: all 10 conversations. Example: 0 runs the first conversation only; 0,1 is the same 2-conversation scope as locomo-mini.
  • --judge enables category-5 judging. Default: disabled.
  • --judge-model MODEL enables category-5 judging and uses MODEL. Default when judge is enabled: gpt-5.4-mini-2026-03-17, unless overridden by BENCH_JUDGE_MODEL.
  • --output PATH saves JSON results to PATH. Default: no output file is written.
  • --recall-limit N sets memories recalled per question. Default: 10.
  • --live runs against Railway instead of local Docker. Default: local Docker.
  • --no-cleanup keeps benchmark test data after the run. Default: cleanup enabled.

make bench-eval wraps the snapshot-based benchmark flow and always writes results to benchmarks/results/<bench>_<config>_<timestamp>.json.

If the judge is disabled, category 5 remains N/A. If the judge is enabled but evidence is missing or the LLM response is invalid, the affected category-5 questions are skipped rather than counted wrong.

What is LoCoMo?

LoCoMo evaluates AI systems' ability to remember and reason across very long conversations (300+ turns). It measures performance across 5 categories:

  1. Single-hop Recall (Category 1) - Simple fact retrieval: "What is Caroline's identity?"
  2. Temporal Understanding (Category 2) - Time-based queries: "When did Caroline move to Sweden?"
  3. Multi-hop Reasoning (Category 3) - Connecting multiple memories: "What fields would Caroline pursue in education?"
  4. Open Domain (Category 4) - General knowledge questions
  5. Complex Reasoning (Category 5) - Advanced inference tasks

Historical note: older public LoCoMo references such as CORE's 88.24% are still useful background context, but they are not AutoMem's primary comparison target because the public setups are not perfectly apples-to-apples, especially around category-5 handling.

AutoMem currently publishes the following LoCoMo baselines:

SetupScopeScoreNotes
locomo, canonical judge on (gpt-5.4-mini-2026-03-17)Full 10 conversations84.74% (1683/1986)Current canonical full run from the May 2026 publication verification; category 5 scored 95.52% (426/446) with 0 skips/errors.
locomo, historical judge on (gpt-5.1)Full 10 conversations83.99% (1668/1986)Historical #128 full run; category 5 scored 92.83% (414/446) with 0 skips.
locomo-mini, judge off2 conversations, categories 1-4 only89.27% (208/233)Historical mini anchor after evaluator fixes; not the current headline full-run claim.
locomo, judge on (gpt-4o)Full 10 conversations87.56% (1739/1986)Historical March 2026 run; kept for trend context only.

Running the Benchmark

# Quick commands
make test-locomo              # Run locally against Docker
make test-locomo-live         # Run against Railway deployment

# With options
./test-locomo-benchmark.sh --recall-limit 20 --output results.json
./test-locomo-benchmark.sh --live --no-cleanup

What the Benchmark Tests

  1. Memory Storage: Loads ~10,000 dialog turns from 10 conversations
  2. Hybrid Recall: Tests semantic + keyword + tag-based retrieval
  3. Graph Relationships: Evaluates multi-hop reasoning via relationship traversal
  4. Temporal Queries: Tests time-based memory filtering
  5. Answer Accuracy: Checks if recalled memories contain correct answers

Performance Expectations

The benchmark takes approximately:

  • Local Docker: 10–20 minutes
  • Railway: 15–25 minutes (network latency)

Memory usage:

  • FalkorDB: ~6,000–10,000 nodes
  • Qdrant: ~6,000–10,000 vectors (1024 dimensions by default with voyage-4; older/OpenAI setups may use 768 or 3072)

Interpreting Results

Example benchmark output:

📊 FINAL RESULTS
🎯 Overall Accuracy: 84.74% (1683/1986)
⏱️ Total Time: 3164s
💾 Total Memories Stored: 5882

📈 Category Breakdown excerpt:
  Complex Reasoning        : 95.52% (426/446)

See benchmarks/EXPERIMENT_LOG.md for the full per-category table. If you run without the judge, category 5 will show as N/A.

Current baselines and methodology notes live in benchmarks/EXPERIMENT_LOG.md.

Current-State Recall Smoke

Use make bench-current-state as the no-LLM regression gate for recall state semantics. It covers active, expired, future, archived, invalidated, evolved, contradictory, replacement, and tag-gated replacement cases, plus the public state_mode=current|history contract.

The smoke target is deterministic and runs against unit fixtures, so it is appropriate for every recall contract change. Use LongMemEval or LoCoMo only when the change affects ranking, scoring, entity expansion, or answer quality.

LongMemEval Benchmark

AutoMem also includes a LongMemEval harness for milestone validation against the ICLR 2025 long-term memory dataset. Treat prefix slices as smoke tests only; cite the canonical full run or the stratified representative mini for current results.

# Representative judged mini: stratified 5 questions per question type (30 total)
make bench-mini-longmemeval

# Prefix smoke only; biased by dataset order
./test-longmemeval-benchmark.sh --max-questions 20

# Full local run
make test-longmemeval

# Full live Railway run
make test-longmemeval-live

Answer-construction prompt variants are harness-only experiments. The evidence-answer config enables an explicit evidence-first answer prompt for #200 probes; it does not change /recall, graph expansion, vector retrieval, or production API behavior. As of the 2026-07-07 mini A/B, evidence-answer trailed the same-day temporal-answer control, so it is not a promoted LongMemEval default.

Current LongMemEval results:

SetupScopeScoreRetrievalNotes
longmemeval-mini representative30 questions, stratified 5 per question type70.00% (21/30)recall@5 96.67% (29/30)Fresh publication verification run: gpt-5-mini answerer, script LLM eval, 2 empty-answer warnings.
longmemeval full canonical500 questions87.00% (435/500)recall@5 97.00% (485/500)Fresh publication verification run: gpt-5-mini answerer, gpt-5.4-mini-2026-03-17 judge, judge_errors=0, memory_ingest_failures=0, publishable=true.
longmemeval full historical500 questions86.20% (431/500)recall@5 97.20% (486/500)April 2026 canonical milestone with the same answerer and judge policy; kept for trend context.
longmemeval partial legacy prefix (50q)50 questions, single-session-user type82.0% (41/50)recall@5 92.0% (46/50)Provisional prefix run with legacy gpt-4o answerer; recall_limit=10, no entity/relation expansion. Not reproduced by the current stratified bench-mini-longmemeval target and not directly comparable to the older 35.6% / 500-question setup.

For future published LongMemEval results, use the pinned judge policy in docs/BENCHMARK_JUDGE_POLICY.md so runs remain comparable over time. The primary answerer is gpt-5-mini; gpt-4o is legacy continuity only and should be labeled as such. Result metadata distinguishes the answer model (answerer_model / llm_model) from the judge model (judge_model, currently gpt-5.4-mini-2026-03-17 when --llm-eval is enabled).

BEAM Benchmark

BEAM experiments currently live in automem-evals, because the harness is exploratory and wraps the upstream memory-benchmarks runner through an AutoMem shim. The durable findings are summarized here because official benchmark claims and docs references belong in this repo.

Current BEAM 100K diagnostic results:

SetupScopeScoreNotes
V1 raw-dialogue shim20 conversations, 400 questions76.25% (305/400), avg 0.677gpt-5-mini answerer/judge, top-k 200, zero errors.
V2 fact-extraction shim20 conversations, 400 questions73.75% (295/400), avg 0.653-2.50pp vs V1 overall. Abstention and knowledge_update improved; event_ordering and information_extraction regressed.

Do not read the BEAM 100K number as an apples-to-apples product comparison with mem0's published 1M/10M results. The 100K tier is an easier retrieval setting, and the shim/judge setup differs from the published comparison target. It is useful as a failure-mode and regression signal.

Benchmark Ownership

automem is the canonical home for official benchmark harnesses, published baselines, and any benchmark numbers referenced in docs, CI, or release notes.

Use the separate automem-evals repo for:

  • ruleset experimentation
  • scenario authoring and seeded corpora
  • cross-agent or cross-backend comparisons
  • bulky timestamped result artifacts and exploratory writeups

Memora/FAMA and WRIT runs remain diagnostic unless their scenario, harness, and result artifacts are promoted into this repo's official benchmark flow and recorded in benchmarks/EXPERIMENT_LOG.md.

If you are building an external eval harness against local AutoMem, use the contract in EVALS_CONTRACT.md and treat this repo as the system under test.

AutoMem's Advantages

AutoMem is expected to perform well due to:

  1. Richer Graph: 11 relationship types, beyond simple temporal-link-only baselines

    • RELATES_TO, LEADS_TO, OCCURRED_BEFORE
    • PREFERS_OVER, EXEMPLIFIES, CONTRADICTS
    • REINFORCES, INVALIDATED_BY, EVOLVED_INTO
    • DERIVED_FROM, PART_OF

  2. Hybrid Search: Vector + keyword + tags + importance + time

    • Designed to recover both semantic matches and explicit structured context
    • Useful for testing graph/vector tradeoffs against snapshot-based evals

  3. Background Intelligence:

    • Entity extraction for structured queries
    • Pattern detection for common themes
    • Consolidation for improved relevance

  4. Dual Storage: FalkorDB + Qdrant provides redundancy and complementary retrieval

Benchmark Setup

The LoCoMo benchmark is automatically cloned during first run:

tests/benchmarks/locomo/
├── data/
│   └── locomo10.json          # 10 conversations, 1,986 questions
├── task_eval/                 # Evaluation utilities
└── README.MD                  # Benchmark documentation

Troubleshooting

"LoCoMo dataset not found"

cd tests/benchmarks
git clone https://github.com/snap-research/locomo.git

Low accuracy scores

  • Check if enrichment pipeline is enabled
  • Verify OpenAI API key is set (for embeddings)
  • Increase --recall-limit (default: 10)
  • Review individual question results in output JSON

Timeout errors

  • Reduce batch size in config
  • Increase pause between batches
  • Use Railway for better performance

Research Citation

@article{maharana2024evaluating,
  title={Evaluating very long-term conversational memory of llm agents},
  author={Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei},
  journal={arXiv preprint arXiv:2402.17753},
  year={2024}
}

Best Practices

  1. Always run unit tests before committing
  2. Run integration tests when changing API logic or database operations
  3. Run live tests before deploying to verify no regressions
  4. Check test coverage with pytest --cov (requires pytest-cov)
  5. Review test output - integration tests show actual API responses
  6. Run LoCoMo benchmark before major releases to validate memory performance