README.org

January 16, 2026 · View on GitHub

  • konserve :PROPERTIES: :CUSTOM_ID: h:6f85a7f4-3694-4703-8c0b-ffcc34f2e5c9 :END:

[[https://clojurians.slack.com/archives/CB7GJAN0L][https://img.shields.io/badge/slack-join_chat-brightgreen.svg]] [[https://clojars.org/org.replikativ/konserve][https://img.shields.io/clojars/v/org.replikativ/konserve.svg]] [[https://circleci.com/gh/replikativ/konserve][https://circleci.com/gh/replikativ/konserve.svg?style=shield]] [[https://github.com/replikativ/konserve/tree/development][https://img.shields.io/github/last-commit/replikativ/konserve/main.svg]]

[[https://whilo.github.io/old/articles/16/unified-storage-io][Simple durability, made flexible.]]

Heads-up: Breaking change konserve 0.9. now requires a UUID under :id in the configuration in general. See below.*

** What is konserve?

A simple document store protocol defined with synchronous and [[https://github.com/clojure/core.async][core.async]] semantics to allow Clojuresque collection operations on associative key-value stores, both from Clojure and ClojureScript for different backends. Data is generally serialized with [[https://github.com/edn-format/edn][edn]] semantics or, if supported, as native binary blobs and can be accessed similarly to =clojure.core= functions =get-in=, =assoc-in= and =update-in=. =update-in= especially allows to run functions atomically and returns old and new value. Each operation is run atomically and must be consistent (in fact ACID), but further consistency across keys is, depending on the backend, only optionally supported.

*** Key Features

  • /cross-platform/ between Clojure and ClojureScript
  • /lowest-common denominator interface/ for an associative datastructure with =edn= semantics
  • /thread-safety with atomicity over key operations/
  • /fast serialization/ options (fressian, transit, ...), independent of the underlying kv-store
  • /very low overhead/ protocol, including direct binary access for high throughput
  • /no additional dependencies and setup/ required for IndexedDB in the browser and the file backend on the JVM and Node.js
  • /avoids blocking io/, the filestore for instance will not block any thread on reading

** Quick Start

Add to your dependencies: [[https://clojars.org/org.replikativ/konserve][https://img.shields.io/clojars/v/org.replikativ/konserve.svg]]

#+BEGIN_SRC clojure (require '[konserve.core :as k])

;; All stores require a UUID :id for global identification ;; Generate once: (java.util.UUID/randomUUID) or (random-uuid) ;; Then use the literal in your config (def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"})

;; Create new store, pass opts as separate argument (def store (k/create-store config {:sync? true}))

;; Use the store (k/assoc-in store [:user] {:name "Alice"} {:sync? true}) (k/get-in store [:user] nil {:sync? true}) ;; => {:name "Alice"}

(k/update-in store [:user :age] (fnil inc 0) {:sync? true}) ;; => [nil 1]

;; Clean up (k/delete-store config) #+END_SRC

** Store Identification (UUID Requirement) :PROPERTIES: :CUSTOM_ID: h:uuid-requirement :END:

All konserve stores require a globally unique =:id= field containing a UUID type. This ensures stores can be uniquely identified and matched across different backends, machines, and synchronization contexts.

Why UUID IDs are required:

  • Global identifiability: Match stores regardless of backend type or file path
  • Cross-machine sync: Identify the same logical store across different systems
  • High entropy: 128-bit UUIDs prevent collisions
  • Backend-agnostic: Same ID works for memory, file, S3, Redis, etc.

How to use UUIDs:

#+BEGIN_SRC clojure ;; 1. Generate a UUID once (in your REPL or terminal) (java.util.UUID/randomUUID) ;; Clojure (random-uuid) ;; ClojureScript ;; => #uuid "550e8400-e29b-41d4-a716-446655440000"

;; 2. Copy the UUID and use it as a literal in your config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"}

;; 3. Pass opts as separate argument to store functions (k/create-store config {:sync? true})

;; 4. Use the SAME UUID every time for the same logical store ;; 5. Use DIFFERENT UUIDs for different stores (dev, test, prod) #+END_SRC

Important:

  • Generate a UUID once and use it consistently for the same store
  • Store the UUID in your application config (EDN files support =#uuid= literals)
  • Different stores (dev, test, prod) should have different UUIDs
  • Never generate UUIDs dynamically in your code - use fixed literals

** Installation

Add to your =deps.edn=:

#+BEGIN_SRC clojure {:deps {org.replikativ/konserve {:mvn/version "LATEST"}}} #+END_SRC

Or to your =project.clj=:

#+BEGIN_SRC clojure [org.replikativ/konserve "LATEST"] #+END_SRC

** Core Concepts

*** Synchronous vs Asynchronous

Konserve supports both synchronous and asynchronous execution modes via =core.async=.

Synchronous mode (=:sync? true=):

#+BEGIN_SRC clojure (def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"})

(def store (k/create-store config {:sync? true}))

(k/assoc-in store [:key] "value" {:sync? true}) (k/get-in store [:key] nil {:sync? true}) ;; => "value" #+END_SRC

Asynchronous mode (=:sync? false=):

#+BEGIN_SRC clojure (require '[clojure.core.async :refer [go <!]])

(def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440001"})

(go (def store (<! (k/create-store config {:sync? false})))

(<! (k/assoc-in store [:key] "value"))
(println (<! (k/get-in store [:key]))))

;; => "value" #+END_SRC

*** Store Lifecycle

Konserve provides five key lifecycle functions:

  • =create-store= - Create a new store, errors if already exists
  • =connect-store= - Connect to existing store, errors if doesn't exist
  • =store-exists?= - Check if store exists at the given configuration
  • =release-store= - Release connections and resources held by a store
  • =delete-store= - Delete underlying storage

#+BEGIN_SRC clojure (def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-446655440002" :path "/tmp/my-store"})

;; Check if store exists (k/store-exists? config {:sync? true}) ;; => false

;; Create new store (errors if already exists) (def store (k/create-store config {:sync? true}))

;; Use the store... (k/assoc-in store [:data] {:value 42} {:sync? true})

;; Later, connect to existing store (errors if doesn't exist) ;; (def store (k/connect-store config {:sync? true}))

;; Clean up resources (k/release-store config store {:sync? true})

;; Delete underlying storage (k/delete-store config {:sync? true})

;; Verify deletion (k/store-exists? config {:sync? true}) ;; => false #+END_SRC

*** Create vs Connect Semantics

All backends follow consistent strict semantics:

Strict semantics (All backends: File, S3, DynamoDB, Redis, LMDB, RocksDB, IndexedDB, Memory with :id):

  • =create-store= - Creates new store, errors if already exists
  • =connect-store= - Connects to existing store, errors if doesn't exist
  • =store-exists?= - Checks for existence before create/connect

** Built-in Backends

*** Memory Store

An in-memory store wrapping an Atom, available for both Clojure and ClojureScript.

#+BEGIN_SRC clojure (require '[konserve.core :as k])

;; Persistent registry-based store with ID (def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440003"})

(def my-db (k/create-store config {:sync? true}))

;; Later sessions can reconnect: ;; (def my-db (k/connect-store config {:sync? true})) #+END_SRC

*** File Store (JVM)

A file-system store using [[https://github.com/clojure/data.fressian][fressian]] serialization. No setup or additional dependencies needed.

#+BEGIN_SRC clojure (require '[konserve.core :as k])

(def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-446655440004" :path "/tmp/konserve-store"})

;; Create new store (def my-db (k/create-store config {:sync? true}))

;; Or connect to existing ;; (def my-db (k/connect-store config {:sync? true})) #+END_SRC

The file store supports:

  • Optional fsync control via =:sync-blob? false= for better performance
  • Custom =java.nio.file.FileSystem= instances via =:filesystem= parameter
  • Thoroughly tested using [[https://github.com/google/jimfs][Jimfs]] (Google's in-memory NIO filesystem)

*** File Store (Node.js)

For Node.js environments, require the Node.js-specific file store:

#+BEGIN_SRC clojure (require '[konserve.core :as k] '[konserve.node-filestore] ;; Registers :file backend for Node.js '[clojure.core.async :refer [go <!]])

(go (def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-446655440005" :path "/tmp/konserve-store"})

(def my-db (<! (k/create-store config {:sync? false}))))

#+END_SRC

*** IndexedDB (Browser)

[[https://developer.mozilla.org/en-US/docs/IndexedDB][IndexedDB]] backend for ClojureScript browser applications. Async-only, must be explicitly required.

#+BEGIN_SRC clojure (require '[konserve.core :as k] '[konserve.indexeddb] ;; Register :indexeddb backend '[clojure.core.async :refer [go <!]])

(go (def config {:backend :indexeddb :id #uuid "550e8400-e29b-41d4-a716-446655440006" :name "my-app-db"})

;; Create new store
(def my-idb-store (<! (k/create-store config {:sync? false})))

;; Use the store
(<! (k/assoc-in my-idb-store [:user] {:name "Alice" :age 30}))
(<! (k/get-in my-idb-store [:user]))

;; Multi-key atomic operations
(<! (k/multi-assoc my-idb-store {:user1 {:name "Alice"}
                                 :user2 {:name "Bob"}}))

;; Efficient bulk retrieval - returns sparse map of found keys
(<! (k/multi-get my-idb-store [:user1 :user2 :nonexistent]))
;; => {:user1 {:name "Alice"} :user2 {:name "Bob"}}

;; Atomic bulk delete
(<! (k/multi-dissoc my-idb-store [:user1 :user2]))

;; Clean up
(<! (k/delete-store config {:sync? false})))

#+END_SRC

The IndexedDB implementation supports atomic multi-key operations through IndexedDB's native transaction model.

** External Backends

External backends integrate seamlessly through the unified store interface. After requiring a backend module, it automatically registers with the multimethod dispatch system.

*** Usage Pattern

#+BEGIN_SRC clojure (require '[konserve.core :as k]) (require '[konserve-s3.core]) ;; Registers :s3 backend

(def config {:backend :s3 :id #uuid "550e8400-e29b-41d4-a716-446655440007" :bucket "my-bucket" :region "us-east-1"})

;; Create new store (def s3-store (k/create-store config {:sync? true}))

;; Use the store (k/assoc-in s3-store [:data] {:value 42} {:sync? true})

;; Later, connect to existing store ;; (def s3-store (k/connect-store config {:sync? true}))

;; Clean up (k/delete-store config {:sync? true}) #+END_SRC

*** Available External Backends

*** Unofficial Backends

*** Outdated Backends

The following projects are incompatible with the latest konserve release, but describe the usage of the underlying store API:

** Advanced Features

*** Tiered Storage

Konserve supports tiered storage with a frontend cache layer and backend persistence layer. Combines a fast frontend store (e.g., in-memory) with a durable backend store (e.g., filesystem).

#+BEGIN_SRC clojure (require '[konserve.core :as k])

(def config {:backend :tiered :id #uuid "550e8400-e29b-41d4-a716-446655440008" :frontend-config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440009"} :backend-config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-44665544000a" :path "/tmp/store"} :write-policy :write-through :read-policy :frontend-first})

;; Create tiered store (creates both frontend and backend) (def tiered-store (k/create-store config {:sync? true}))

;; Use the store (k/assoc-in tiered-store [:data] {:value 42} {:sync? true})

;; Clean up (k/delete-store config {:sync? true}) #+END_SRC

Write policies:

  • =:write-through= - Write to backend, then frontend synchronously
  • =:write-around= - Write only to backend, invalidate frontend

Read policies:

  • =:frontend-first= - Check frontend first, fallback to backend (populates frontend)
  • =:frontend-only= - Only read from frontend

The tiered store supports multi-key operations (=multi-get=, =multi-assoc=, =multi-dissoc=) when both stores support them. During initialization, =multi-get= combined with =multi-assoc= enables efficient bulk sync from backend to frontend.

*** Multi-Key Operations

Multi-key operations provide atomic bulk operations for supported backends:

#+BEGIN_SRC clojure (require '[konserve.core :as k])

;; Check if backend supports multi-key operations (k/multi-key-capable? store) ;; => true/false

;; Atomic bulk write (k/multi-assoc store {:user1 {:name "Alice"} :user2 {:name "Bob"} :user3 {:name "Carol"}} {:sync? true})

;; Efficient bulk read - returns sparse map (only found keys) (k/multi-get store [:user1 :user2 :nonexistent] {:sync? true}) ;; => {:user1 {:name "Alice"} :user2 {:name "Bob"}}

;; Atomic bulk delete (k/multi-dissoc store [:user1 :user2] {:sync? true}) #+END_SRC

Backends with multi-key support:

  • Memory store
  • IndexedDB
  • Tiered store (when both layers support it)

*** Write Hooks

Write hooks are invoked after every successful write operation, enabling reactive patterns like store synchronization, change logging, or triggering side effects.

#+BEGIN_SRC clojure (require '[konserve.core :as k])

(def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-44665544000b"})

(def store (k/create-store config {:sync? true}))

;; Register a hook to log all writes (k/add-write-hook! store ::my-logger (fn [{:keys [api-op key value]}] (println "Write:" api-op key "->" value)))

;; Writes now trigger the hook (k/assoc-in store [:user] {:name "Alice"} {:sync? true}) ;; Prints: Write: :assoc-in :user -> {:name "Alice"}

;; Remove hook when done (k/remove-write-hook! store ::my-logger) #+END_SRC

Hook function receives:

  • =:api-op= - The operation (=:assoc-in=, =:update-in=, =:dissoc=, =:bassoc=, =:multi-assoc=, =:multi-dissoc=)
  • =:key= - The top-level key being written
  • =:key-vec= - Full key path (for =assoc-in= / =update-in=)
  • =:value= - The value written
  • =:old-value= - Previous value (for update operations)
  • =:kvs= - Map of key->value (for =multi-assoc=)
  • =:keys= - Collection of keys (for =multi-dissoc=)

Hooks are invoked at the API layer (in =konserve.core=), so they work consistently across all store backends. Stores must implement the =PWriteHookStore= protocol.

*** Garbage Collection

Konserve has a garbage collector that can be called manually when the store gets too crowded.

#+BEGIN_SRC clojure (require '[konserve.gc :as gc])

;; Evict keys older than cutoff date, keep whitelisted keys (gc/sweep! store cutoff-date whitelist {:sync? true}) #+END_SRC

The function =konserve.gc/sweep!= allows you to provide a cut-off date to evict old keys and a whitelist for keys that should be kept.

*** Compression and Encryption

Compression and encryption are supported by the default store implementation used by all current backends except lmdb and memory.

#+BEGIN_SRC clojure ;; Store configuration with compression and encryption (def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-44665544000c" :path "/tmp/secure-store" :config {:encryptor {:type :aes :key "s3cr3t"} :compressor {:type :lz4}}})

(def store (k/create-store config {:sync? true})) #+END_SRC

Compression:

  • LZ4 compression (JVM only)

Encryption:

  • AES/CBC/PKCS{5/7}Padding with 256 bit
  • Different salt for each written value
  • Same cold storage format for JVM and JS (cross-runtime compatible)

*** Serialization Formats

Different formats for =edn= serialization like [[https://github.com/clojure/data.fressian][fressian]], [[http://blog.cognitect.com/blog/2014/7/22/transit][transit]] or a simple =pr-str= version are supported and can be combined with different stores. Stores have a reasonable default setting. You can extend the serialization protocol to other formats if needed. [[https://github.com/replikativ/incognito][Incognito]] support is available for custom records.

**** Tagged Literals

You can read and write custom records according to [[https://github.com/replikativ/incognito][incognito]].

*** Error Handling

For synchronous execution, normal exceptions are thrown. For asynchronous error handling, we follow the semantics of =go-try= and =<?= introduced [[https://swannodette.github.io/2013/08/31/asynchronous-error-handling][here]].

The [[https://github.com/replikativ/superv.async/][superv.async]] library provides error handling for core.async. You just need two macros: =<?= checks for an exception and rethrows, =go-try= catches and passes it along as a return value so errors don't get lost.

** Backend Implementation Guide :PROPERTIES: :CUSTOM_ID: h:7582b1c9-e305-4d51-a808-c10eb447f3de :END:

We provide a [[file:doc/backend.org][backend implementation guide]].

New in 2025: External backends can register with the unified store dispatch system by defining multimethod implementations for:

  • =konserve.store/create-store= - Create new store, error if exists
  • =konserve.store/connect-store= - Connect to existing store, error if doesn't exist
  • =konserve.store/store-exists?= - Check if store exists
  • =konserve.store/delete-store= - Delete underlying storage
  • =konserve.store/release-store= - Release resources

All backends must implement strict semantics where =create-store= errors if the store already exists and =connect-store= errors if the store doesn't exist. See existing external backends (konserve-s3, konserve-lmdb, konserve-rocksdb, konserve-redis, konserve-dynamodb) for reference implementations.

** Projects Building on Konserve :PROPERTIES: :CUSTOM_ID: h:79876ac1-414b-4180-8d65-63737cb3bc53 :END:

** Combined Usage with Other Writers :PROPERTIES: :CUSTOM_ID: h:8a1b4a06-4b9f-496b-9eb2-52ac953a8e35 :END:

Konserve assumes it accesses its keyspace in the store exclusively. It uses [[https://github.com/replikativ/hasch][hasch]] to support arbitrary edn keys and hence does not normally clash with outside usage even when the same keys are used. To support multiple konserve clients in the store, the backend must support locking and proper transactions on keys internally, which is the case for backends like CouchDB, Redis and Riak.

** License :PROPERTIES: :CUSTOM_ID: h:8153b6f6-d253-4863-86b4-038dd383b6fe :END:

Copyright © 2014-2026 Christian Weilbach and contributors

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.