Synchronization

Barrel replicates documents between databases with a version-vector protocol: causal ordering from a hybrid logical clock (HLC), deterministic winners, idempotent delivery, and checkpoints for resumption. Replication runs in the same VM or over HTTP against a barrel_server. Read this when you need to copy or keep two databases in sync, replicate a subset with channels, or move attachments along with documents.

What it is

Replication lives in barrel_docdb (barrel_rep). One run:

1. Read the checkpoint (a local doc on both sides) to find the last shipped HLC.
2. Fetch changes from the source since that HLC, filter applied at the source.
3. Diff the offered version tokens against the target in one round trip.
4. put_version the missing documents on the target (idempotent, so at-least-once delivery is safe).
5. Checkpoint per batch, then run the attachment phase (see below).

Both sides converge to the same winner, body, and deletion state for every document, whatever the interleaving of writes. Conflicts are recorded where the concurrency was observed and resolve by writing a superseding version.

Transports implement the barrel_rep_transport behaviour. Two ship today: barrel_rep_transport_local (same VM, database names) and barrel_rep_transport_http (hackney client speaking the /db/:db/_sync/* wire served by barrel_server).

When to use it

• Copy a database to another database, in the same VM or on another node.
• Keep replicas eventually consistent, one-shot per run or continuously.
• Replicate a subset of documents with a channel, path, or query filter.
• Move attachment blobs with their documents, content-addressed.

How (same VM)

{ok, _} = barrel_docdb:create_db(<<"source">>),
{ok, _} = barrel_docdb:create_db(<<"target">>),
{ok, _} = barrel_docdb:put_doc(<<"source">>, #{<<"id">> => <<"a">>, <<"v">> => 1}),

{ok, Result} = barrel_rep:replicate(<<"source">>, <<"target">>),
#{docs_written := N, att_sync := _} = Result.

Run it again later to pick up new changes; it resumes from the checkpoint.

How (over HTTP)

The remote side is a URL under a running barrel_server. Build an endpoint, then pass the matching transport:

Endpoint = barrel_rep_transport_http:endpoint(
    <<"http://edge-1.example.com:8080/db/inventory">>),

%% push
{ok, _} = barrel_rep:replicate(<<"inventory">>, Endpoint,
    #{target_transport => barrel_rep_transport_http}),

%% pull
{ok, _} = barrel_rep:replicate(Endpoint, <<"inventory">>,
    #{source_transport => barrel_rep_transport_http}).

endpoint/1 normalizes the URL (scheme and host lowercased, no trailing slash). The normalized URL is the replication identity: changing its text starts a fresh checkpoint, credentials and tuning do not. The endpoint map also takes pool, connect_timeout, recv_timeout, and headers.

How (auth)

Protect a server with static bearer tokens; a list accepts old and new during a rotation. /health stays open, everything else requires the token:

%% server side (app env, before barrel_server starts)
application:set_env(barrel_server, auth, #{tokens => [<<"s3cret">>]}).

Give the client its token on the endpoint, or keyed by origin in the app env so persisted task configs stay secret-free:

Authed = Endpoint#{auth => #{token => <<"s3cret">>}},
%% or
application:set_env(barrel_docdb, sync_auth,
                    #{<<"http://edge-1.example.com:8080">> => <<"s3cret">>}).

How (signed requests)

Bearer tokens replay without TLS. For per-request Ed25519 authentication, add accept (the list of accepted methods) and signers (keyId to 32-byte raw public key). Each request is signed over ts|keyId|nonce|METHOD|target|sha256(body), where nonce is fresh per request and target is the path plus ? and the raw query string when there is one (signature v2), with replay protection and a skew window. accept => [bearer, signed] accepts both, so a fleet rolls over node by node; accept => [bearer] (or no accept) is exactly the bearer-only behavior above.

%% server: know each peer's public key
application:set_env(barrel_server, auth,
                    #{accept  => [bearer, signed],
                      signers => #{<<"node1">> => PubKey32},
                      skew_ms => 300000}).

%% client: sign with the matching private key (endpoint or origin env)
Signed = Endpoint#{signing => #{key_id => <<"node1">>, priv_key => Priv32}},
%% or
application:set_env(barrel_docdb, sync_signing,
                    #{<<"https://edge-1.example.com:8443">> =>
                          #{key_id => <<"node1">>, priv_key => Priv32}}).

Generate a key pair with crypto:generate_key(eddsa, ed25519). Signing takes precedence over a bearer token on the same endpoint. Private keys are never written into persisted task configs.

Rolling from signature v1 (barrel_docdb < 1.4.0, no nonce, path only) to v2: servers on barrel_server 1.6.0+ accept both, so upgrade every server first, then the clients, which send v2 only (a v2 client against an older server is refused with 401 on every signed request). Once no client sends v1 (v1 acceptances log at debug level), close the v1 window:

application:set_env(barrel_server, auth,
                    #{accept => [signed], signers => Signers,
                      require_nonce => true}).

A reverse proxy in front of the sync endpoint must forward the path and the query string unmodified: the query is part of the signed target.

How (TLS and mTLS)

barrel_server serves cleartext HTTP/1.1 by default. Configure listeners to add HTTP/2 and HTTP/3, and TLS. Setting verify => verify_peer makes a listener a mutual-TLS gate: a client without a CA-signed certificate is refused at the handshake. The replication client speaks HTTP/1.1, so run the sync gate on an H1-over-TLS listener (tls => true on the http entry).

application:set_env(barrel_server, listeners,
                    #{http  => #{port => 8443, tls => true},  %% H1 over TLS
                      https => #{port => 8444},               %% H2
                      http3 => #{port => 8444}}),             %% H3
application:set_env(barrel_server, tls,
                    #{certfile => "server.pem", keyfile => "server.key",
                      cacertfile => "ca.pem", verify => verify_peer}),
application:set_env(barrel_server, auth, #{accept => [mtls]}).

%% client: present its cert (endpoint or origin env)
Mtls = Endpoint#{ssl_options => [{certfile, "client.pem"},
                                 {keyfile, "client.key"},
                                 {cacertfile, "ca.pem"}]}.

Notes: H1-TLS and H2 are full mTLS gates (OTP ssl). H3 serves over TLS but is not yet a client-cert gate (a QUIC-level change is pending); do not rely on H3 for mTLS.

What mTLS authenticates: that the client holds a certificate chaining to the configured CA, not which client it is. accept => [mtls] alone treats every such client as one trusted peer with access to every database. For identity and per-database rights, layer bearer or signed auth on top (accept => [mtls, signed] gates the transport and identifies the peer by its signing key). Mapping a client certificate to an identity needs livery to surface the peer certificate, which it does not yet.

How (selective replication)

Filters apply at the source. Path and query filters compose with AND:

{ok, _} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{
        paths => [<<"users/#">>],
        query => #{where => [{path, [<<"status">>], <<"active">>}]}
    }
}).

Channels are the indexed alternative: named pattern sets declared at database creation, materialized into a per-channel feed at write time, so a filtered pull is one bounded scan instead of a full feed walk:

{ok, _} = barrel_docdb:create_db(<<"source">>, #{
    channels => #{<<"mobile">> => [<<"type/task">>, <<"owner/+">>]}
}),
{ok, _} = barrel_rep:replicate(<<"source">>, <<"target">>, #{
    filter => #{channel => <<"mobile">>}
}).

Channel notes:

• Patterns are MQTT style (+ one segment, trailing # for the rest).
• Channels are fixed at creation and index writes made after creation.
• A document that stops matching stops being sent; replicas keep their last copy (no target-side delete). A fresh replica never sees departed docs.
• A filtered replication keeps its own checkpoint (the filter joins the replication id).

Attachments

Attachment sync rides every run (attachments => true by default). It is a second phase after the document loop: the source's attachment feed is digest-diffed against the target, only missing content transfers, and writes converge last-write-wins on their origin HLC. The result lands under att_sync in the replication result:

{ok, #{att_sync := #{atts_written := _, atts_skipped := _}}} =
    barrel_rep:replicate(<<"source">>, Endpoint,
        #{target_transport => barrel_rep_transport_http}).

Notes:

• Blobs stream in chunks both directions over HTTP; nothing buffers a blob whole. Pulls have no size ceiling. Pushes are bounded by the server's {barrel_server, max_body, Bytes} env (default 1 GiB, infinity accepted); a larger push gets a 413, fails for that attachment only, and the run reports it in att_write_failures.
• There is no ordering between documents and their attachments: a doc can arrive before its blob (reads 404 until the attachment phase lands).
• Attachments written before this feature do not sync until rewritten, or synthesize feed rows once with barrel_docdb:rebuild_attachment_feed/1.
• Turn the phase off with attachments => false; it degrades to att_sync => skipped on its own when a side cannot track attachments.

Continuous replication (tasks)

The task manager (barrel_rep_tasks) persists replications across restarts. Remote ends are URLs; the HTTP transport is picked automatically:

{ok, TaskId} = barrel_rep_tasks:start_task(#{
    source => <<"inventory">>,
    target => <<"http://edge-1.example.com:8080/db/inventory">>,
    mode => continuous,          %% or one_shot
    direction => push,           %% push | pull | both
    filter => #{channel => <<"mobile">>}
}),
{ok, #{status := running}} = barrel_rep_tasks:get_task(TaskId),
ok = barrel_rep_tasks:stop_task(TaskId).

Continuous behavior:

• Local sources are event driven: the task wakes on the changes stream and drains through its filter, so local convergence is tens of milliseconds.
• Remote sources poll adaptively, 500 ms after data and backing off to 15 s while idle.
• Transient errors do not kill a continuous task: it backs off (1 s to 60 s, with jitter), records last_error on the task doc, and stays running. One-shot tasks fail fast.
• Task docs store endpoints as URLs only; tokens come from sync_auth.

Clocks

Every wire exchange carries an x-barrel-hlc header and both sides fold it into their clock, on top of the explicit handshake at the start of each run. A peer too far ahead is rejected with {error, clock_skew} (409 on the wire). To couple clocks by hand:

{ok, _Merged} = barrel_docdb:sync_hlc(RemoteHlc).

Browser clients

The same /db/:db/_sync/* wire drives barrel-lite, the TypeScript browser client: it keeps an offline-first OPFS store, stamps local mutations with its own source id, and pushes and pulls over this protocol (adaptive polling, not SSE). Enable CORS and issue a capability token for it. See barrel-lite.

What is not covered yet

• TLS serving. barrel_server listens on plain HTTP; the client can reach https servers. Put a TLS terminator in front for now.
• Vectors. Vector indexes are not shipped; with record mode the text and metadata replicate as documents and the index rebuilds locally. Quantized vector sync is planned with the TypeScript client.
• Barrel API. barrel (the full embeddable database) does not expose replication; call barrel_rep and barrel_rep_tasks on the underlying docdb name.
• Provenance. The actor/session/source a write carries (see audit-provenance) does not travel on the wire. Replicated arrivals are attributed cause: replicated with the origin database's identity; the acting agent stays on the origin's own trail.