MemPalace.NET

April 24, 2026 · View on GitHub

Overview

MemPalace.NET uses a pluggable backend architecture for storing embeddings and metadata. The default backend is SQLite-based, offering a zero-dependency local storage solution with brute-force vector similarity search.

Backend Interface

All backends must implement the IBackend and ICollection interfaces defined in MemPalace.Core.Backends.

IBackend

The backend is responsible for:

  • Creating and managing palace databases
  • Creating, listing, and deleting collections within palaces
  • Enforcing embedder identity consistency
  • Health checking

Key methods:

ValueTask<ICollection> GetCollectionAsync(
    PalaceRef palace,
    string collectionName,
    bool create = false,
    IEmbedder? embedder = null,
    CancellationToken ct = default);

ValueTask<IReadOnlyList<string>> ListCollectionsAsync(
    PalaceRef palace, 
    CancellationToken ct = default);

ValueTask DeleteCollectionAsync(
    PalaceRef palace, 
    string name, 
    CancellationToken ct = default);

ValueTask<HealthStatus> HealthAsync(CancellationToken ct = default);

ICollection

The collection is responsible for:

  • CRUD operations on embedded records
  • Vector similarity search (cosine distance)
  • Metadata filtering via WhereClause DSL
  • Dimension and embedder identity validation

Key methods:

ValueTask AddAsync(IReadOnlyList<EmbeddedRecord> records, CancellationToken ct = default);
ValueTask UpsertAsync(IReadOnlyList<EmbeddedRecord> records, CancellationToken ct = default);
ValueTask<GetResult> GetAsync(IReadOnlyList<string>? ids = null, WhereClause? where = null, ...);
ValueTask<QueryResult> QueryAsync(IReadOnlyList<ReadOnlyMemory<float>> queryEmbeddings, int nResults = 10, ...);
ValueTask<long> CountAsync(CancellationToken ct = default);
ValueTask DeleteAsync(IReadOnlyList<string>? ids = null, WhereClause? where = null, ...);

Error Handling

Backends must throw these specific exceptions:

  • PalaceNotFoundException — palace does not exist when create=false
  • EmbedderIdentityMismatchException — embedder identity doesn't match collection
  • DimensionMismatchException — embedding dimensions don't match collection
  • UnsupportedFilterException — backend cannot handle a specific WhereClause type
  • BackendClosedException — backend has been disposed

Default: SQLite Backend

Package: MemPalace.Backends.Sqlite
Dependencies: Microsoft.Data.Sqlite (9.0.0)

Architecture

  • One database per palace: Each PalaceRef maps to a palace.db file in {LocalPath}/palace.db or {BaseDirectory}/{PalaceId}/palace.db if no LocalPath is provided.
  • One table per collection: Collections are stored in tables named collection_{name}.
  • Metadata table: _meta table stores embedder identity and dimensionality per collection.

Schema

Metadata Table

CREATE TABLE _meta (
    collection_name TEXT PRIMARY KEY,
    embedder_identity TEXT NOT NULL,
    dimensions INTEGER NOT NULL
)

Collection Table

CREATE TABLE [collection_{name}] (
    id TEXT PRIMARY KEY,
    document TEXT NOT NULL,
    metadata TEXT NOT NULL,    -- JSON
    embedding BLOB NOT NULL,    -- float32 array as bytes
    dim INTEGER NOT NULL
)

Vector Storage

Current implementation: Embeddings are stored as BLOBs (byte arrays of float32 values) and searched using brute-force cosine similarity in C#.

Why brute-force?

  • Simple, reliable, no external dependencies
  • Sufficient for collections up to ~100K records on modern hardware
  • Zero setup overhead

Future options:

  • sqlite-vec extension: Native vector search with HNSW indexing. Currently not available as a stable NuGet package, but could be integrated when available.
  • Microsoft.SemanticKernel.Connectors.Sqlite: Heavier dependency but Microsoft-stewarded. Considered overkill for MemPalace's needs.

Cosine Distance Computation

distance = 1 - (dot_product / (magnitude_a * magnitude_b))

Lower distances indicate higher similarity. Results are sorted ascending by distance.

Filter Translation

WhereClause objects are translated to SQL using SQLite's json_extract function:

Eq("tag", "test")       → json_extract(metadata, '$.tag') = 'test'
Gt("count", 5)          → json_extract(metadata, '$.count') > 5
In("status", [1, 2])    → json_extract(metadata, '$.status') IN (1, 2)
And([clause1, clause2]) → (clause1) AND (clause2)

Supported operators: Eq, NotEq, Gt, Gte, Lt, Lte, In, NotIn, And, Or.

Unsupported clauses throw UnsupportedFilterException.

Usage

using MemPalace.Backends.Sqlite;

var backend = new SqliteBackend("/path/to/palaces");
var palace = new PalaceRef("my-palace");
var embedder = ...; // your IEmbedder implementation

var collection = await backend.GetCollectionAsync(
    palace, 
    "documents", 
    create: true, 
    embedder: embedder);

// Add records
var records = new[] { 
    new EmbeddedRecord("id1", "hello world", metadata, embedding1),
    ...
};
await collection.AddAsync(records);

// Query
var queryResults = await collection.QueryAsync(
    new[] { queryEmbedding }, 
    nResults: 10);

// Dispose
await backend.DisposeAsync();

Writing a Custom Backend

  1. Create a new project referencing MemPalace.Core.
  2. Implement IBackend and ICollection.
  3. Handle all required exceptions (PalaceNotFoundException, etc.).
  4. Pass BackendConformanceTests: 
    public class MyBackendConformanceTests : BackendConformanceTests
{
    protected override IBackend CreateBackend() => new MyBackend();
}
  5. Document vector search strategy (exact, approximate, hybrid).
  6. Document filter support (which WhereClause types are handled).

Example: Qdrant Backend Stub

public class QdrantBackend : IBackend
{
    private readonly QdrantClient _client;

    public QdrantBackend(string url) 
    { 
        _client = new QdrantClient(url); 
    }

    public async ValueTask<ICollection> GetCollectionAsync(...)
    {
        // Map palace.Id + collectionName to Qdrant collection
        var qdrantCollectionName = $"{palace.Id}_{collectionName}";
        
        if (!await _client.CollectionExistsAsync(qdrantCollectionName))
        {
            if (!create) throw new PalaceNotFoundException(...);
            await _client.CreateCollectionAsync(qdrantCollectionName, ...);
        }

        return new QdrantCollection(_client, qdrantCollectionName, embedder);
    }

    // ... implement other methods
}

Conformance Testing

All backend implementations should pass the BackendConformanceTests suite in MemPalace.Tests. This ensures:

  • Correct CRUD behavior
  • Upsert idempotence
  • Query ordering (by distance, ascending)
  • Filter operators (Eq, And, etc.)
  • Dimension and embedder identity guards
  • Collection lifecycle (list, delete)
  • Backend state management (closed state handling)

Run tests:

dotnet test --filter "FullyQualifiedName~BackendConformanceTests"
``$

## \text{Performance} \text{Considerations}

**\text{SQLite} \text{Backend}:**
- **\text{Reads}:** \text{O}(\text{n}) \text{for} \text{queries} (\text{full} \text{scan} \text{with} \text{filtering})
- **\text{Writes}:** \text{O}(1) \text{per} \text{record} (\text{indexed} \text{by} \text{ID})
- **\text{Space}:** ~4  \times  \text{dimensions}  \times  \text{record\_count} \text{bytes} \text{for} \text{embeddings}

\text{For} \text{large} \text{collections} (>100\text{K} \text{records}) \text{or} \text{latency}-\text{critical} \text{queries}, \text{consider}:
- **\text{Qdrant}** — \text{distributed}, \text{HNSW}, \text{GPU}-\text{accelerated}
- **\text{Chroma}** — \text{Python}-\text{first}, \text{good} \text{for} \text{hybrid} \text{search}
- **\text{Pinecone}** — \text{managed}, \text{serverless}, \text{high}-\text{scale}

\text{All} \text{of} \text{these} \text{can} \text{be} \text{integrated} \text{by} \text{implementing} \text{the} $IBackend` interface.

## Summary

- **Default:** SQLite with brute-force cosine similarity
- **Pluggable:** Implement `IBackend` + `ICollection` for custom backends
- **Tested:** All backends must pass `BackendConformanceTests`
- **Scalable:** Swap to Qdrant/Chroma/Pinecone as needed

For questions or contributions, see the main [README](../README.md).