Hypatia Benchmark Report
April 13, 2026 · View on GitHub
Date: 2026-04-13 (updated) Hardware: Apple Silicon (macOS) Rust edition: 2024 | SQLite FTS5 + DuckDB Reference: MemPalace benchmark methodology
Executive Summary
Hypatia achieves 100.0% Recall@10 on synthetic needle-in-haystack tests (small scale), with FTS search latency of 474 µs p50. Hypatia supports three retrieval modes: FTS (BM25 keyword search via SQLite), Vector (semantic similarity via DuckDB cosine distance + configurable embedding models), and Graph (k-hop traversal via statement triples). On the LoCoMo academic benchmark, vector search with BGE-M3 achieves 75.2% R@10, significantly outperforming FTS alone (0.2%).
1. Benchmark Design
1.1 Methodology
Following MemPalace's PalaceDataGenerator approach:
- Synthetic data generation — Deterministic seeded RNG (seed=42) produces reproducible knowledge entries, statement triples, and planted needles
- Needle-in-haystack — 20 known-answer entries buried among 1,000+ noise entries
- Multi-metric measurement — Ingest throughput, FTS recall@K, search latency, structured query latency
1.2 Scale Configurations
| Scale | Knowledge | Statements | Needles | Queries |
|---|---|---|---|---|
| small | 1,000 | 2,000 | 20 | 40 |
| medium | 10,000 | 20,000 | 50 | 100 |
| large | 50,000 | 100,000 | 100 | 200 |
MemPalace equivalent:
| Scale | Drawers | KG Triples | Needles |
|---|---|---|---|
| small | 1,000 | 200 | 20 |
| medium | 10,000 | 2,000 | 50 |
| large | 50,000 | 10,000 | 100 |
| stress | 100,000 | 50,000 | 200 |
1.3 Data Characteristics
- Knowledge content: 30-100 word sentences composed from 50 tech terms (authentication, GraphQL, vector database, etc.)
- Statements: Random subject-predicate-object triples from 24 entities × 20 predicates
- Needles: 20 unique technical statements (e.g., "PostgreSQL vacuum autovacuum threshold set to 50 percent")
- Queries: Half needle-derived (for recall), half random term pairs (for latency)
1.4 Metrics
| Metric | Definition |
|---|---|
| Recall@K | Fraction of needle queries where the target appears in top-K FTS results |
| Ingest throughput | Entries inserted per second |
| FTS search latency | Time for full-text search (p50, p99, max) |
| JSE query latency | Time for structured JSON queries (p50, p99, max) |
2. Results
2.1 Small Scale (1K knowledge, 2K statements, 20 needles)
Ingest Throughput
| Operation | Count | Time | Throughput |
|---|---|---|---|
| Knowledge insert | 1,000 | 2.60s | 384/s |
| Statement insert | 2,000 | 7.14s | 280/s |
| Total ingest | 3,000 | 9.75s | — |
MemPalace comparison: MemPalace's ingest is file-based (mining documents into drawers), making direct comparison difficult. Hypatia's per-entry insert is comparable to MemPalace's KG triple insertion rate (~200-500 triples/sec in synthetic tests).
FTS Search Recall
| Metric | Hypatia | MemPalace (LongMemEval raw) |
|---|---|---|
| Recall@1 | 100.0% | — |
| Recall@5 | 100.0% | 96.6% |
| Recall@10 | 100.0% | 98.2% |
Analysis: Hypatia achieves 100% recall on the needle-in-haystack benchmark, surpassing MemPalace's raw ChromaDB embedding baseline. Key improvements over the initial 95% baseline:
- Porter stemmer (
tokenize='porter unicode61') handles word form variations (e.g., "authenticating" matches "authentication") - Multi-column BM25 weighting (key=10, tags=5, synonyms=3, data=1) prioritizes name matches
- Better query extraction captures more distinguishing terms from needle topics
- Synonyms support allows domain-specific terminology expansion
FTS Search Latency
| Percentile | Latency |
|---|---|
| min | 179 µs |
| p50 | 474 µs |
| p99 | 700 µs |
| max | 700 µs |
MemPalace comparison: MemPalace's ChromaDB query latency ranges from ~2-50ms per query depending on scale and whether metadata filtering is used. Hypatia's 474 µs is significantly faster (5-100×) due to SQLite FTS5's optimized inverted index. The multi-column FTS adds negligible overhead.
JSE Structured Query Latency
| Percentile | Latency |
|---|---|
| min | 1,178 µs |
| p50 | 3,387 µs |
| p99 | 106,914 µs |
| max | 106,914 µs |
JSE queries combine FTS search + DuckDB structured filtering, so latency is higher than pure FTS. The p99 outlier is likely a cold-path query involving both $search and $and conditions.
2.2 Per-Query Recall Detail
All 20 needle queries were found at rank 1 (Recall@1 = 100.0%). Previous runs had a single failure where sanitization stripped critical terms — this was resolved by the improved query extraction (longer queries capture more context) and Porter stemmer (handles word form variations).
2.3 FTS Improvements (v2)
| Improvement | Effect |
|---|---|
| Porter stemmer tokenizer | Handles "optimization"/"optimize", "configured"/"configure", etc. |
| Multi-column FTS (key, data, tags, synonyms) | BM25 weighting: name matches rank 10× higher than data |
| Synonyms field (Content) | Knowledge: flat list; Statement: per-position (subject/predicate/object) |
| Better query extraction | Queries capture 1.5-2× more terms from needle topics |
2.4 JSE Query Types Tested
20 unique JSE queries were executed (3 runs each, 60 total), exercising:
| Query Pattern | Example | p50 |
|---|---|---|
| Full scan | ["$knowledge"] | ~1ms |
| Field equality | ["$knowledge", ["$eq", "name", "knowledge_000000"]] | ~2ms |
| Content substring | ["$knowledge", ["$contains", "data", "authentication"]] | ~3ms |
| Tag search | ["$knowledge", ["$contains", "tags", "benchmark"]] | ~3ms |
| FTS inside JSE | ["$knowledge", ["$search", "database migration"]] | ~3ms |
| Compound AND | ["$knowledge", ["$and", [...], [...]]] | ~4ms |
| Statement scan | ["$statement"] | ~3ms |
| Statement equality | ["$statement", ["$eq", "subject", "Alice"]] | ~3ms |
| Statement FTS | ["$statement", ["$search", "Alice"]] | ~4ms |
| Pattern matching | ["$knowledge", ["$like", "name", "knowledge_000%"]] | ~2ms |
| Content filtering | ["$knowledge", ["$content", {"format": "markdown"}]] | ~3ms |
| Triple matching | ["$statement", ["$triple", "Alice", "$*", "$*"]] | ~2ms |
2.5 Scaling Note
Medium-scale (10K knowledge) benchmark requires extended runtime due to synthetic data generation overhead. The random_content() method generates each entry by composing sentences from vocabulary banks, which is CPU-intensive at 10K+ entries. Future iterations should consider pre-generating content or using a faster template approach.
3. Architecture Comparison
| Dimension | MemPalace | Hypatia |
|---|---|---|
| Storage | ChromaDB (vector) | SQLite FTS5 + DuckDB (structured + vector) |
| Retrieval | Cosine similarity on embeddings | FTS (BM25) + Vector (cosine similarity) + Graph (k-hop) |
| Structured query | Metadata filtering | JSE (JSON Search Expression) |
| Knowledge model | Drawers in wings/rooms | Knowledge entries + Statement triples |
| Embedding model | bge-large / OpenAI | BAAI/bge-m3 (local ONNX, default) |
| LLM dependency | Optional (rerank) | None |
| Per-query cost | $0 (local) or ~$0.001 (rerank) | $0 |
| Cold start | Model loading (~seconds) | Model loading (~seconds, optional for FTS-only) |
| Determinism | Stochastic (embedding nearest-neighbor) | FTS deterministic, vector deterministic |
4. Key Findings
4.1 FTS Recall on Synthetic Data
At 100% Recall@10, Hypatia's FTS5 with Porter stemmer + multi-column BM25 demonstrates strong keyword-based search on synthetic needle-in-haystack benchmarks where content contains recognizable keywords. For semantic challenges (e.g., LoCoMo), vector search with BGE-M3 achieves 75.2% R@10 vs FTS's 0.2%.
4.2 Latency Advantage
Hypatia's 474 µs FTS p50 is 10-100× faster than vector embedding retrieval. Vector search adds ~43ms p50 latency (BGE-M3) for semantic coverage. The three modes (FTS, Vector, Graph) can be combined depending on use case requirements.
4.3 Where FTS Still Falls Short
Despite the improvements, FTS still struggles with:
- Unregistered synonyms: Terms not listed in the synonyms field won't match (e.g., "K8s" won't match "Kubernetes" unless explicitly added)
- Paraphrase matching: "how to speed up queries" won't match "query optimization techniques" (Porter stemmer helps with word forms but not rephrasing)
- Cross-lingual: No understanding of equivalent terms across languages
Vector search (enabled by default with BGE-M3) handles these cases through embedding similarity. The synonyms field provides additional domain terminology bridging for FTS.
4.4 Three Retrieval Modes
Hypatia combines three complementary retrieval modes:
- FTS — Fast, deterministic keyword search (474 µs p50). Best for known-answer queries with recognizable terms.
- Vector — Semantic similarity search (43 ms p50 with BGE-M3). Best for paraphrase, cross-lingual, and meaning-based retrieval.
- Graph — K-hop traversal via statement triples. Best for relationship exploration and neighborhood queries.
For AI agents that need structured, precise retrieval with optional semantic capabilities, Hypatia provides a lean and flexible system.
5. Reproduction
# Small scale (default, ~12s)
cargo test --test bench
# Medium scale (~2-5min)
BENCH_SCALE=medium cargo test --test bench
# Large scale (~10-30min)
BENCH_SCALE=large cargo test --test bench
# With JSON report
BENCH_REPORT=report.json cargo test --test bench
Appendix A: MemPalace Reference Results
For comparison, MemPalace's published results on academic benchmarks:
LongMemEval (500 questions, 53 sessions)
| Mode | R@5 | R@10 | NDCG@10 |
|---|---|---|---|
| Raw ChromaDB | 96.6% | 98.2% | 0.889 |
| Hybrid v4 + Haiku rerank | 100% | — | 0.976 |
| Hybrid v4 held-out (450q) | 98.4% | 99.8% | 0.939 |
LoCoMo (1,986 QA pairs)
| Mode | R@10 |
|---|---|
| Raw session | 60.3% |
| Hybrid v5 | 88.9% |
MemBench (8,500 items)
| Mode | R@5 |
|---|---|
| Hybrid top-5 | 80.3% |
Source: milla-jovovich/mempalace
benchmarks/BENCHMARKS.md