hako

August 3, 2026 · View on GitHub

Clojars Project

A Modern, high-performance, Low-alloc Binary Serialization Library for Clojure.

Built on JDK 25 FFM MemorySegment. Quite possibly the fastest — and lowest-allocation — binary serializer for Clojure on the JVM today. See Benchmarks.

Highlights

  • Zero runtime dependencies — only org.clojure/clojure.
  • Off-heap by default — MemorySegment via JDK 25 FFM.
  • Low GC pressure — arena buffer reused across messages, segment-out path allocates ~zero per call.
  • Tunable — reusable Writer/Reader, caller-owned arenas, zero-copy byte decode, prim-array packing.
  • Java hot path — per-value dispatch and primitive writes in Java
  • Per-message symbol table — repeated keywords / symbols dedup to a 1-byte symref.
  • Secure by default — no class loading from wire, no Serializable, no decompression. See Security.
  • Extensible — records (Clojure + Java), user-tag registry with length-prefixed frames for forward-compatible reads.

Status

Pre-release, alpha.

Requirements

  • JDK 25+ — uses java.lang.foreign FFM API. Requires --enable-native-access=ALL-UNNAMED on the JVM CLI (the packaged jar bundles the Enable-Native-Access manifest entry so application consumers don't see the warning).
  • Clojure 1.12+ — decoder probes PersistentArrayMap threshold at load, adapts to 1.13's bumped keyword-only limit automatically.

Install

Clojars Project

Quick start

(require '[s-exp.hako :as hako])

(def bs (hako/encode {:name "Alice" :tags #{:a :b :c} :score 42}))
(hako/decode bs)
;; => {:name "Alice", :tags #{:a :b :c}, :score 42}

Note. hako/encode and hako/decode are drop-in helpers to ease migration from other serializers — each call allocates a fresh Writer / Reader (~500 B / ~250 B) and opens a private confined Arena. They are not the idiomatic way to use hako. Reach for encode-into! / decode-into! (reusable Writer / Reader) or the ThreadLocal-pooled variants encode-pooled / decode-pooled for hot paths. See the API section below and Performance.

API

Encoding

;; Migration-friendly one-shots — convenient, but allocate a fresh
;; Writer (~500 B) and confined Arena per call. Prefer the reusable
;; or pooled forms below on any hot path.
(hako/encode value)                    ; -> byte[]
(hako/encode value opts)               ; -> byte[]

(hako/encode-to-segment arena value)   ; -> MemorySegment (caller owns arena)
(hako/encode-to-segment arena value opts)

;; Idiomatic — reusable writer for high-throughput encode loops:
(with-open [wr (hako/writer 4096)]
  (dotimes [_ 1000]
    (let [seg (hako/encode-into! wr some-value)]
      ;; consume `seg` before the next call — the slice is
      ;; overwritten on the next encode-into!
      ...)))

;; Zero-alloc write straight into a caller-owned buffer (byte[] or
;; ByteBuffer). Skips the MemorySegment.asSlice wrapper — useful for
;; direct-to-LMDB / socket / mmap paths.
(hako/encode-into-buffer! wr dst value)   ; -> byte count written

;; Batch API — multiple values share one symbol table:
(hako/encode-many [{:a 1} {:a 2} {:a 3}])
;; keyword :a is encoded once, symref'd twice.

;; Batch encode into a reusable writer (skips the ~500 B Writer setup
;; that `encode-many` pays per call):
(hako/encode-many-into! wr [{:a 1} {:a 2} {:a 3}])   ; -> MemorySegment

;; Pooled convenience — ThreadLocal-backed writer, opt-in:
(hako/encode-pooled value)                ; -> byte[], zero setup alloc after warmup

;; Transducer-friendly decode over a batch stream:
(into #{} (filter :active) (hako/decoder bs))
(sequence (map :id) (hako/decoder bs))
(reduce + 0 (hako/decoder bs))

Encode options

OptionDefaultDescription
:initial-size256Starting buffer size in bytes.
:preserve-metafalsePreserve metadata on IObj values via the with-meta extension tag.
:pack-homogeneousfalseDetect all-Long / all-Double vectors and emit them as packed prim arrays.
:coerce-custom-comparatorfalseAllow sorted-set-by / sorted-map-by — the custom comparator is dropped on decode.

Decoding

(hako/decode src)                      ; src is byte[] or MemorySegment
(hako/decode src opts)

;; Reusable reader:
(let [rd (hako/reader some-src)]
  (hako/decode-into! rd another-src))

;; Pooled convenience — ThreadLocal-backed reader, opt-in.
;; Skips the ~250 B per-call Reader alloc.
(hako/decode-pooled bs)

;; Cleanup for short-lived / servlet-container threads that use the
;; pooled variants. Closes the pooled Writer's confined Arena and
;; drops scratch buffers. Long-lived pooled threads never need this.
(hako/close-thread-locals!)

;; Batch — inverse of encode-many, returns a vector of values:
(hako/decode-many bs)

Decode options

OptionDefaultDescription
:zero-copyfalseReturn MemorySegment slices for byte payloads instead of copying to byte[].
:tolerate-unknown-tagsfalseUnregistered user-tag ids yield TaggedValue instead of throwing.
:cache-identsfalseConsult a JVM-global cache when interning decoded keywords / symbols.

Supported types

Semantic equality (=) is preserved for all listed types.

  • nil, boolean, Character, Long, Integer, Short, Byte, Double, Float, String.
  • byte[], long[], double[], int[], float[], short[], char[], boolean[] — packed, component type preserved.
  • Object[] (and any reference-typed array — component type not preserved, decodes as Object[]).
  • Keyword, Symbol — with per-message symbol table + symref dedup.
  • UUID, java.util.Date, java.util.regex.Pattern (flags preserved), java.net.URI.
  • java.time: Instant, Duration, Period, LocalDate, LocalTime, LocalDateTime, ZonedDateTime, OffsetDateTime.
  • BigInteger, clojure.lang.BigInt, BigDecimal, Ratio.
  • PersistentVector, PersistentList, PersistentHashSet, PersistentHashMap, PersistentArrayMap, ISeq.
  • PersistentTreeSet, PersistentTreeMap — default comparator only; custom comparators cause a write error unless :coerce-custom-comparator true.
  • PersistentQueue.
  • Clojure records via defrecord (requires registration).
  • Java records (JEP 395; requires registration).
  • User-defined types via ext/register-user-tag!.

Concrete map / set impls may differ across Clojure versions — the reader picks PersistentArrayMap vs PersistentHashMap based on the runtime's threshold. See SPEC.md §5 for the roundtrip contract.

Extensions

Records

(require '[s-exp.hako.ext :as ext])

(defrecord Point [x y])
(ext/register-record! Point)

(hako/decode (hako/encode (->Point 3 4)))
;; => #user.Point{:x 3, :y 4}

Registration reflects on the class once and caches a MethodHandle for the canonical positional constructor.

Java records work identically:

public record Point(int x, int y) {}

(ext/register-record! com.example.Point)

User-tagged types

(import '(java.io File))

(ext/register-user-tag!
 1                                      ; small app-local id (shifted to 0x10000001 on wire)
 File
 (fn write [w f] (.writeString w (.getPath ^File f)))
 (fn read  [r]
   (let [tag (.getByte r)
         low (bit-and tag 0x0F)
         n (.readTierPayload r (int low))]
     (File. (.getString r (int n))))))

(hako/decode (hako/encode (File. "/tmp/data.edn")))
;; => #object[java.io.File ...]

Frames are length-prefixed, so an unknown user-tag id can be skipped by a :tolerate-unknown-tags reader without derailing the surrounding message. See EXTENSIONS.md §E.2.

Metadata

Opt-in per encode:

(hako/encode (with-meta [1 2 3] {:tag :vec})
             {:preserve-meta true})

Wire format

Byte-level spec in SPEC.md. Extension registry in EXTENSIONS.md. Byte-by-byte worked examples in WIRE_EXAMPLES.md. Highlights:

  • 5-byte envelope <magic 'HAKO'><version 0>.
  • Every value starts with a tag byte: high nibble = major type, low nibble = size tier or subtype.
  • Fixed-width size tiers (inline 0..11, u8, u16, u32, u64) — no varint on the hot path.
  • Little-endian throughout.
  • Per-message symbol table for interned keyword / symbol payloads.
  • Zero shared state across messages.

Security

hako is designed for decoding untrusted input safely. Key guarantees:

  • No arbitrary class loading. Records only instantiate classes registered via ext/register-record! — the wire carries a classname string, but hako looks it up in the registry rather than calling Class.forName. An attacker cannot force instantiation of arbitrary Java classes.
  • No arbitrary code execution via user-tags. User-tag ids dispatch through the register-user-tag! registry. Unregistered ids throw by default; :tolerate-unknown-tags true returns an opaque TaggedValue{:ext id :bytes segment-slice} — never invokes unknown code.
  • No Java Serializable fallback. hako has no path to ObjectInputStream. Deserialization gadget chains are not applicable.
  • No decompression. The wire format doesn't ship compressed payloads — no zip / gzip / snappy decompression on the read path, so no compression-bomb amplification vector.
  • Per-message symbol table. Interning state is scoped to one message. A malicious message can't poison state for future decodes.
  • Bounded reads. Count and length fields are validated against remaining segment bytes before allocation. Silent truncation for u64-tier counts that exceed Integer/MAX_VALUE is rejected cleanly, not truncated.
  • Envelope enforcement. Magic + version bytes are checked before any dispatch.
  • Confined memory. Encoder writes into Arena.ofConfined() — cross-thread misuse is blocked by the FFM layer with WrongThreadException, not a memory corruption.

Not defended against (out of scope):

  • Malicious user-tag write / read callbacks you register yourself. Registered code runs with your JVM's privileges — vet the callbacks you install.
  • Denial-of-service via extreme payload sizes. hako reads what you give it; enforce input size limits at the transport layer.

Benchmarks

Criterium (quick-bench statistics: 6 samples × 100 ms target, 1.5 s JIT warmup per call), JDK 25, -server -Xmx4g, direct-linking on. Single machine, AC power — reproduce with clj -M:bench -m bench.

hako's core value proposition is off-heap encoding into a MemorySegment with minimal Java-heap allocation. Four call styles are measured; encode-into! uses a long-lived reusable Writer (arena buffer amortized across messages), the others open a fresh arena per call:

  • encode-into! → seg — reusable Writer, emits a MemorySegment slice per message. No byte[] allocation per call, no arena open/close per call. The differentiator — this is what the design is optimized for.
  • encode-into! → byte[] — same reusable Writer, then a trailing MemorySegment → byte[] copy for callers stuck on byte[] APIs.
  • encode → byte[] — one-shot hako/encode. Fresh confined arena + final off-heap → heap copy per call. Convenient, less optimal.
  • encode-to-segment → seg — one-shot with a caller-provided Arena. Arena setup cost per call, but the result stays off-heap in the caller's arena.

Peers (JVM-only, on-heap output): nippy (default freeze, compression + checksums), nippy-fast (fast-freeze, no compression), deed (com.github.igrishaev/deed-core 0.1.0), and transit (com.cognitect/transit-clj 1.0.333, MsgPack — different niche, reference only).

Encode — hako call-style ladder

Absolute times per call. encode-into! → seg is the ceiling; other columns show what each additional convenience costs.

payloadencode-into! →segencode-into! →byte[]encode →byte[]encode-to-segment →seg
long-array-1k0.20 µs0.72 µs0.87 µs0.64 µs
double-array-1k0.19 µs0.71 µs0.84 µs0.61 µs
string-1000.03 µs0.04 µs0.09 µs0.14 µs
small-map0.11 µs0.11 µs0.19 µs0.26 µs
mixed0.20 µs0.22 µs0.29 µs0.39 µs
string-10k0.74 µs1.07 µs1.30 µs1.23 µs
vec-of-strings1.31 µs1.30 µs1.57 µs1.64 µs
nested-map (50 kw)3.44 µs3.45 µs4.32 µs4.33 µs
vec-of-longs (1k)5.74 µs5.89 µs6.32 µs6.42 µs

The segment-out path wins by ~4.3× on prim arrays, ~1.8× on long strings, and matches the byte[] paths on collection payloads (where per-value dispatch dominates the arena/copy costs). Numbers are the mean of two full runs; hako cells reproduce within a few percent, peer-library outliers (e.g. nippy on string-10k) vary more across runs.

Encode — encode-into! →seg vs peers

payloadencode-into! →segnippynippy-fastdeedtransitvs nippy-fast
long-array-1k0.20 µs18.4 µs18.3 µs11.1 µs22.1 µs91×
double-array-1k0.19 µs22.5 µs10.8 µs10.9 µs24.1 µs58×
string-1000.03 µs0.12 µs0.07 µs0.42 µs3.0 µs2.2×
small-map0.11 µs0.31 µs0.26 µs0.61 µs3.6 µs2.4×
mixed0.20 µs0.56 µs0.52 µs0.85 µs4.2 µs2.6×
string-10k0.74 µs2.73 µs1.13 µs2.12 µs4.5 µs1.5×
vec-of-strings1.31 µs2.33 µs2.47 µs3.61 µs6.96 µs1.9×
nested-map (50 kw)3.44 µs10.1 µs10.1 µs16.1 µs36.3 µs2.9×
vec-of-longs (1k)5.74 µs17.8 µs17.6 µs21.6 µs31.5 µs3.1×

encode-into! →seg leads every cell — including string-10k where the byte[]-output paths lose to nippy-fast.

Decode

Decode has two hako variants — one-shot hako/decode (byte[] source, wrapped via MemorySegment/ofArray internally) and reused hako/decode-into! (segment source, no wrap per call). Both take {:cache-idents true}.

payloaddecode-into! →seg srcdecode →byte[] srcnippynippy-fastdeedtransitvs nippy-fast
long-array-1k0.58 µs0.59 µs13.8 µs13.5 µs10.6 µs194 µs23×
double-array-1k0.58 µs0.58 µs12.2 µs8.2 µs10.8 µs177 µs14×
string-1000.04 µs0.05 µs0.09 µs0.05 µs0.56 µs2.8 µs1.0×
small-map0.15 µs0.25 µs0.24 µs0.18 µs0.79 µs3.3 µs1.2×
mixed0.32 µs0.28 µs0.67 µs0.64 µs1.3 µs4.6 µs2.0×
string-10k0.88 µs1.05 µs3.6 µs1.1 µs1.71 µs5.70 µs1.2×
vec-of-strings2.64 µs2.68 µs3.83 µs3.07 µs8.24 µs16.4 µs1.2×
nested-map (50 kw)6.12 µs5.93 µs14.4 µs14.3 µs25.8 µs55.2 µs2.3×
vec-of-longs (1k)11.43 µs11.39 µs14.0 µs13.7 µs25.9 µs194 µs1.2×

decode-into! →seg src leads every cell; string-100 is a dead heat with nippy-fast (~1 ns apart) — its readUTF intrinsic is hard to beat on payloads smaller than a cache line. See Performance for the tradeoffs.

Allocation

Bytes allocated per operation, measured via ThreadMXBean.getThreadAllocatedBytes over 100 000 iterations post-warmup. Lower = less GC pressure on the consuming system. Reproduce with clj -M:bench -m alloc-bench.

Encode — encode-into! →seg vs nippy/fast-freeze

payloadencode-into! →segnippy-fastvs nippy-fast
`long-array-1k$4037888947 \times \text{less}
$string-10k`10056201522.0× less
`ns-map$ (50 \text{kw})4021784544 \times \text{less}
$string-100`1603522.2× less
small-map882883.3× less
mixed965045.3× less
vec-of-strings244034721.4× less
`nested-map$ (50 \text{kw})4010000250 \times \text{less}

\text{Decode} — $decode-into!→seg src vsnippy/fast-thaw(measured with{:cache-idents true}`)

payloaddecode-into! →seg srcnippy-fastvs nippy-fast
long-array-1k8088709768.8× less
nested-map (50 kw)7792401765.2× less
small-map2408483.5× less
string-10k10112201922.0× less
string-1002163841.8× less
mixed60023443.9× less
ns-map (50 kw)3664248006.8× less
vec-of-strings580882561.4× less

encode-into! →seg wins allocation on every measured cell — encode and decode. Keyword-heavy encode paths (nested-map, ns-map) sit at the 40 B MemorySegment-slice baseline once the Writer is warmed and the global keyword-bytes cache has been populated. Use encode-into-buffer! to hand the encoded bytes directly into a caller-owned byte[] / ByteBuffer — skips the 40 B slice wrapper for callers who own their output buffer.

This allocation delta is the tail-latency story — invisible to mean-of-loop timing benches. Under load (e.g. 100k msg/s), a 10 KB/op reduction per encode is 1 GB/s less young-gen churn on the consuming JVM. See Performance for tuning notes.

Records — 100 records in a vector

5-field Event record ({:id :ts :user :action :payload}), 100 instances in a vector. Reproduce with clj -M:bench -m records-bench.

metricencode-into! / decode-into!nippy-fastmultiplier
encode8.7 µs60 µs6.9×
decode26 µs72 µs2.7×
size2933 B10088 B3.4× smaller

Records are where the per-message symbol table pays off hardest. hako emits the record classname + field-key keywords once per message, then symrefs them (1 byte each) for the remaining 99 records. Nippy re-emits every keyword payload. Result: ~7× faster encode, ~3× faster decode, ~3.4× smaller wire vs nippy-fast. The win grows with vector length as the symref-vs- payload ratio widens.

Nippy's default freeze compresses the output with Snappy and comes in at 2173 B — smaller than hako's uncompressed 2933 B, but at the cost of adding Snappy on the decode path (compression bomb vector). Wrap hako in transport-layer compression if the tradeoff makes sense for your setup.

Registration required — see Extensions §Records.

Encoded size

payloadhakonippynippy-fastdeedtransit
nested-map (50 kw)732 B1632 B1628 B3598 B1128 B
vec-of-longs (1k)2740 B2878 B2874 B10024 B2619 B
long-array-1k8009 B2880 B2876 B8040 B2619 B
double-array-1k8009 B4165 B8997 B8040 B9003 B
vec-of-strings797 B896 B892 B1330 B793 B
mixed46 B55 B51 B154 B54 B
small-map37 B39 B35 B101 B34 B
string-100107 B106 B102 B140 B108 B
string-10k10008 B61 B10003 B10040 B10008 B

Two patterns dominate:

  • Keyword-heavy structures: hako wins big — nested-map (50 kw) is ~55% smaller than nippy. Per-message symbol table compresses repeat idents to 1-byte symrefs; peers re-emit the full ident each time.
  • Homogeneous numeric arrays: nippy wins via varint on long-array-1k / double-array-1k. hako uses fixed 8-byte layout — same as MemorySegment.copy intrinsics, no per-element encode/decode branching on the hot path. The wire loss buys the perf win visible in the encode/decode tables above.
  • Large repetitive strings: nippy compresses string-10k to 61 bytes via Snappy. hako intentionally does not compress — wrap in your transport-layer compression if you need it. Keeps the decode path free of decompression bomb amplification.

Everywhere else, hako is within a few bytes of the best peer.

Reproduce

clj -M:bench -m bench                  # full sweep (~15 min)
clj -M:bench -m bench nested-map       # single payload
clj -M:bench -m quick                  # 5-payload triage bench, ~40 s

Documentation

Full user guides in docs/:

Wire-format specifications:

Other:

Development

clj -T:build javac       # compile Java sources → target/classes
clj -T:build javac-test  # compile Java test-support classes
clj -M:test              # run full test suite (currently 425 assertions)
clj -M:bench -m bench    # criterium benchmarks vs peers (~15min)
clj -M:bench -m quick    # 5-payload triage bench (~40s)
clj -T:build jar         # build the release jar

License

Mozilla Public License 2.0 — see LICENSE.