Storage Backends

Datahike provides pluggable storage through konserve, allowing you to choose the backend that best fits your deployment model and performance requirements.

Quick Reference

Backend	Best For	Distribution	Durability	Write Throughput
File	Unix tools, rsync, git-like workflows	Single machine	High	Good
LMDB	High-performance single machine	Single filesystem	High	Excellent
RocksDB	Embedded high-write workloads	Single machine	High	Excellent
Memory	Testing, ephemeral data	Single process	None	Excellent
JDBC	Existing SQL infrastructure	Multi-machine	High	Good
Redis	High write throughput	Multi-machine	Medium	Excellent
S3	Distributed scale-out, cost-effective	Multi-region	Very high	Good
GCS	Google Cloud scale-out	Multi-region	Very high	Good
Azure Blob	Azure-native object storage	Multi-region	Very high	Good
DynamoDB	Low latency, AWS-native	Multi-region	Very high	Excellent (expensive)
IndexedDB	Browser persistence	Browser	Medium	Good

Local Backends

File Backend

On Windows, the JVM file backend requires JDK 22+ and --enable-native-access=ALL-UNNAMED for Konserve's directory persistence barrier. This does not raise the Unix JVM requirement. Native binary builds use GraalVM 25. Persistence errors fail the write; do not enable unsafe directory-sync fallback to bypass them.

Use when: You want to use Unix tools (rsync, git, backup scripts) to manage your database.

Key advantage: Deltas in persistent data structures translate directly into individual file deltas, making incremental backups and synchronization highly efficient.

{:store {:backend :file
         :path "/var/lib/myapp/db"}}

Characteristics:

• Each immutable index fragment stored as an individual file
• Efficient incremental backups with rsync
• Can version database directories similar to or with git
• Good for single-machine deployments
• Extensively tested and reliable without external dependencies
• Not ideal for databases with a lot of churn

LMDB Backend

Use when: You need maximum performance on a single machine within a single filesystem.

Key advantage: Lightning-fast memory-mapped database with ACID transactions, optimized for read-heavy workloads.

;; Requires: org.replikativ/konserve-lmdb
{:store {:backend :lmdb
         :path "/var/lib/myapp/db"}}

Characteristics:

• Memory-mapped for zero-copy reads
• Single filesystem only (not distributed)
• Excellent read performance
• Lower memory overhead than file backend
• Well suited for very high churn of small changes
• Very low latency
• Large file blob that cannot be as efficiently synched a the file store

Note: The LMDB backend is available as the separate konserve-lmdb library. It is not part of the portable server artifact: it requires Java 22+, native access, and a platform liblmdb installation.

RocksDB Backend

RocksDB is available through konserve-rocksdb. It remains a separate server deployment dependency: its rocksdbjni artifact is about 68 MiB because it carries native libraries for several operating systems and CPU architectures. Bundling it would nearly double the portable server JAR.

Native-store server variants therefore have different contracts from the portable artifact:

• a RocksDB distribution must select and test its target platforms instead of hiding a multi-platform JNI payload in the standard JAR;
• an LMDB distribution must select Java 22+ and supply liblmdb plus native access at launch.

Until those distributions are published, use the thin server artifact from a Clojure project that adds and requires the corresponding Konserve backend.

Memory Backend

Use when: Testing, development, or ephemeral data that doesn't need to survive process restarts.

{:store {:backend :memory
         :id #uuid "550e8400-e29b-41d4-a716-446655440030"}}

Characteristics:

• No persistence - data lost on process exit
• Fastest possible performance
• Ideal for unit tests and REPL development
• Multiple databases distinguished by :id

Distributed Backends

All distributed backends support Distributed Index Space (DIS): multiple reader processes can directly access shared storage without database connections, enabling massive read scalability.

Important: a connection owns its branch head and serializes commits through it. :writer-ownership :exclusive asserts that all of a database's writers live in one JVM (it does not enforce it); under :shared (the default) several processes may write the same branch, protected by fenced head writes on backends that support conditional writes (:require-fencing true refuses to write unfenced elsewhere). Maintenance that runs on the writers' side — notably garbage collection — can see only the commits in flight in its own process, so under shared ownership it needs the sweep floor described there. Readers are unconstrained. A writer endpoint (see Distributed) remains the way to funnel all writes through one process.

JDBC Backend

Use when: You already have PostgreSQL or another JDBC database in your infrastructure.

Key advantage: Leverage existing SQL database skills, backup procedures, and monitoring tools.

;; Requires: org.replikativ/konserve-jdbc
{:store {:backend :jdbc
         :dbtype "postgresql"
         :host "db.example.com"
         :port 5432
         :dbname "datahike"
         :user "datahike"
         :password "..."}}

Characteristics:

• Use familiar SQL database operations
• Existing backup/restore procedures work
• Read scaling via DIS (readers don't interfere with writer in database's MVCC)
• Good for teams already operating PostgreSQL
• Available for: PostgreSQL, MySQL, H2, and others

Note: The standalone server includes konserve-jdbc and the PostgreSQL driver. Embedding applications add the driver for the SQL database they use.

Redis Backend

Use when: You need high write throughput and can tolerate weaker durability guarantees.

Key advantage: Excellent write performance with in-memory speed.

;; Requires: org.replikativ/konserve-redis
{:store {:backend :redis
         :host "redis.example.com"
         :port 6379}}

Characteristics:

• Very high write throughput
• Durability depends on Redis persistence settings (RDB/AOF)
• Can lose recent writes on Redis crash
• Good for high-traffic applications where some data loss is acceptable
• Distributed reads via DIS

S3 Backend

Use when: You want cost-effective distributed storage that scales to massive datasets.

Key advantage: Extremely scalable, pay-per-use pricing, natural fit for cloud-native architectures.

;; Requires: org.replikativ/konserve-s3
{:store {:backend :s3
         :bucket "my-datahike-bucket"
         :region "us-east-1"}}

Characteristics:

• Unlimited scalability
• Very low storage costs (compared to databases)
• High durability (11 nines)
• Eventually consistent (may have slight read lag)
• Ideal for read-heavy workloads with occasional writes
• Can have high latency
• Works well with AWS Lambda deployments

Performance note: Higher latency than local storage, but cost-effective for billions of datoms.

Google Cloud Storage (GCS) Backend

Use when: You're on Google Cloud Platform and want distributed storage.

Key advantage: Similar to S3 but optimized for GCP infrastructure.

;; Requires: org.replikativ/konserve-gcs
{:store {:backend :gcs
         :bucket "my-datahike-bucket"
         :project-id "my-project"}}

Characteristics:

• Similar to S3 in characteristics
• Native GCP integration
• Good latency within GCP regions
• Cost-effective for large datasets

The standalone server carries the Google SDK's default HTTP/JSON client. GCS production endpoints are still HTTPS with the normal Google authentication and integrity checks; only the optional gRPC implementation is omitted.

Azure Blob Storage Backend

Use when: You're on Azure and want distributed object storage.

;; Requires: org.replikativ/konserve-azure-blob
{:store {:backend :azure-blob
         :container "datahike"
         :account-name "my-storage-account"}}

The standalone server uses Azure's portable JDK HTTP provider. It does not bundle the alternative Netty provider or its platform-native TLS binaries.

DynamoDB Backend

Use when: You need low-latency distributed storage and are willing to pay premium pricing.

Key advantage: Single-digit millisecond latency with strong consistency options.

;; Requires: org.replikativ/konserve-dynamodb
{:store {:backend :dynamodb
         :table "datahike"
         :region "us-east-1"}}

Characteristics:

• Very low latency
• Strong consistency available
• Higher costs than S3
• Good for latency-sensitive applications
• On-demand or provisioned capacity modes

Browser Backend

IndexedDB Backend

Use when: Building offline-capable browser applications with persistent local storage.

Key advantage: Durable browser-local storage with ClojureScript support.

;; ClojureScript only
{:store {:backend :indexeddb
         :id "my-app-db"}}

Characteristics:

• Persistent across browser sessions
• ~50MB-unlimited quota (browser-dependent)
• Asynchronous API
• Often paired with TieredStore for performance

Advanced: TieredStore

TieredStore creates memory hierarchies by layering backends, with faster storage in front of slower, more durable storage.

Use cases:

• Browser: Memory (fast) → IndexedDB (persistent)
• Server: Memory → LMDB → S3 (hot → warm → cold)
• AWS: LMDB (fast local) → S3 (distributed backup)

;; Example: Fast memory cache backed by S3
{:store {:backend :tiered
         :id #uuid "550e8400-e29b-41d4-a716-446655440031"
         :frontend-config {:backend :memory
                          :id #uuid "550e8400-e29b-41d4-a716-446655440031"}
         :backend-config {:backend :s3
                         :bucket "persistent-store"
                         :region "us-east-1"
                         :id #uuid "550e8400-e29b-41d4-a716-446655440031"}
         :write-policy :write-through
         :read-policy :frontend-first}}

How it works:

• Reads check tiers in order (cache-first)
• Writes go to all tiers
• Stacking multiple tiers supported but rarely needed
• Provided by konserve's tiered store implementation

Common patterns:

Browser with offline support:

{:store {:backend :tiered
         :id #uuid "550e8400-e29b-41d4-a716-446655440032"
         :frontend-config {:backend :memory
                          :id #uuid "550e8400-e29b-41d4-a716-446655440032"}
         :backend-config {:backend :indexeddb
                         :id #uuid "550e8400-e29b-41d4-a716-446655440032"}
         :write-policy :write-through}}

AWS Lambda with S3 backing:

{:store {:backend :tiered
         :id #uuid "550e8400-e29b-41d4-a716-446655440033"
         :frontend-config {:backend :lmdb
                          :path "/tmp/cache"
                          :id #uuid "550e8400-e29b-41d4-a716-446655440033"}
         :backend-config {:backend :s3
                         :bucket "lambda-data"
                         :region "us-east-1"
                         :id #uuid "550e8400-e29b-41d4-a716-446655440033"}}}

Backend-Specific Configuration

Each backend may have additional configuration options. See the konserve backend documentation for details:

• konserve - Core abstraction
• konserve-lmdb - LMDB implementation
• datahike-lmdb - Datahike LMDB integration
• datahike-jdbc - JDBC backends
• konserve-s3 - S3 backend
• konserve-redis - Redis backend

Choosing a Backend

For Development

→ Memory or File backend for simplicity

For Single-Machine Production

→ LMDB for best performance → File if you need Unix tool integration

For Distributed Production (Read Scaling)

→ S3/GCS for cost-effective scale → DynamoDB for low latency (higher cost) → JDBC if you already operate PostgreSQL

For High Write Throughput

→ Redis if you can tolerate some data loss → LMDB for durable local writes → DynamoDB for distributed writes (expensive)

For Browser Applications

→ IndexedDB for persistence → TieredStore (Memory → IndexedDB) for speed + durability

For Cost Optimization

→ File backend with rsync for cheap backups → S3 for large datasets (pennies per GB) → TieredStore to minimize expensive tier access

Migration Between Backends

To migrate from one backend to another:

1. Export from source database:

(require '[datahike.migrate :refer [export-db import-db]])
(export-db source-conn "/tmp/datoms-export")

2. Create destination database with new backend:

(d/create-database new-config)
(def dest-conn (d/connect new-config))

3. Import into destination:

(import-db dest-conn "/tmp/datoms-export")

The export format (CBOR) preserves all data types including binary data.

Performance Considerations

Read Performance

• Fastest: Memory, LMDB (memory-mapped)
• Fast: File (SSD), Redis
• Good: JDBC, S3 (with tiering)
• Variable: DynamoDB (provisioned vs on-demand)

Write Performance

• Fastest: Memory, Redis
• Fast: LMDB, DynamoDB (provisioned)
• Good: File, JDBC, S3
• Slower: S3 (especially small writes)

Distribution

• No distribution: Memory, File, LMDB (single filesystem)
• Distributed reads: All cloud backends via DIS
• Single writer: All backends (architectural constraint)

Durability

• None: Memory (ephemeral)
• Medium: Redis (depends on persistence settings), IndexedDB
• High: File, LMDB, JDBC
• Very high: S3, GCS, DynamoDB (11 nines)

Custom Backends

Datahike can use any konserve backend. To create a custom backend:

1. Implement the konserve protocols
2. Register your backend with konserve
3. Use it in Datahike configuration

See the konserve documentation for details on implementing custom backends.