Unicity Aggregator

August 26, 2026 · View on GitHub

High-performance aggregator service for the Unicity blockchain platform. Provides JSON-RPC 2.0 API endpoints for state transition aggregation with MongoDB persistence, BFT consensus integration, and high availability support.

Overview

The Unicity Aggregator implements a decentralized Agent-Aggregator communication layer that enables:

State Transition Submission: Agents submit state transition requests to the aggregation layer
Proof Generation: Retrieval of timestamped inclusion proofs and global non-deletion proofs
Block Management: Blockchain block creation and querying capabilities
High Availability: Distributed processing with automatic leader election

Features

✅ JSON-RPC 2.0 API - Complete implementation of all aggregator methods
✅ MongoDB Integration - Efficient storage with proper indexing
✅ High Availability - Leader election and distributed processing
✅ Signature Validation - Full secp256k1 cryptographic validation for commitments
✅ SMT Integration - Sparse Merkle Tree for inclusion proofs
✅ Round Management - Automated block creation with batch processing
✅ Raw v2 Hash Support - 32-byte SHA256 state and transaction hashes
✅ Configurable Concurrency - Request rate limiting and parallel processing
✅ Graceful Shutdown - Proper resource cleanup on termination
✅ Health Monitoring - Comprehensive health and status endpoints
✅ TLS Support - HTTPS/TLS configuration for production
✅ CORS Support - Cross-origin resource sharing for web clients
✅ Performance Testing - Built-in performance test with cryptographically valid data
✅ BFT Consensus - Integration with Unicity BFT network for block finalization

Quick Start

Prerequisites

Go 1.25 or later
MongoDB 4.4 or later
Make (optional, for using Makefile)

Installation

# Clone the repository
git clone https://github.com/unicitynetwork/aggregator-go.git
cd aggregator-go

# Install dependencies
go mod download

# Build the application
make build
# OR
go build -o bin/aggregator ./cmd/aggregator

Quick Start with Docker

The easiest way to get started is using Docker Compose:

# Start all services (MongoDB, BFT nodes, and Aggregator)
docker compose up -d

# Start with clean state (removes old data)
make docker-run-clean

# View logs
docker compose logs -f aggregator

# Stop services
docker compose down

This will start:

MongoDB on localhost:27017 with admin credentials
BFT Root Node - Consensus root node
Aggregator on localhost:3000 with full BFT integration

Basic Usage (Local Development)

# Start MongoDB (if not using Docker)
mongod --dbpath /your/db/path

# Run with default configuration
./bin/aggregator

# Run with custom configuration
export MONGODB_URI="mongodb://localhost:27017"
export PORT="8080"
export LOG_LEVEL="debug"
./bin/aggregator

The service will start on http://localhost:3000 by default.

⚠️ BFT Configuration Required: For the aggregator to run properly, BFT configuration is required. This includes setting up BFT configuration files and bootstrap node addresses. See bft-support.md for detailed setup instructions.

Configuration

The service is configured via environment variables:

Chain Configuration

Variable	Description	Default
`CHAIN_ID`	Chain ID	`unicity`
`CHAIN_VERSION`	Chain version	`1.0`
`CHAIN_FORK_ID`	Chain's Fork ID	`testnet`

Server Configuration

Variable	Description	Default
`PORT`	HTTP server port	`3000`
`HOST`	HTTP server host	`0.0.0.0`
`READ_TIMEOUT`	HTTP read timeout	`30s`
`WRITE_TIMEOUT`	HTTP write timeout	`30s`
`IDLE_TIMEOUT`	HTTP idle timeout	`120s`
`DEFAULT_REQUEST_TTL`	Lifetime assigned to requests that omit `expiresAt`	`1h`
`CONCURRENCY_LIMIT`	Max concurrent requests	`1000`
`ENABLE_DOCS`	Enable /docs endpoint	`true`
`ENABLE_CORS`	Enable CORS headers	`true`
`ENABLE_TLS`	Enable HTTPS/TLS	`false`
`TLS_CERT_FILE`	TLS certificate file path	``
`TLS_KEY_FILE`	TLS private key file path	``
`ENABLE_H2C`	Serve HTTP/2 cleartext (h2c) alongside HTTP/1.1	`true`
`HTTP2_MAX_CONCURRENT_STREAMS`	Max concurrent HTTP/2 streams per connection	`4096`

Database Configuration

Variable	Description	Default
`MONGODB_URI`	MongoDB connection string	`mongodb://localhost:27017`
`MONGODB_DATABASE`	Database name	`aggregator`
`MONGODB_CONNECT_TIMEOUT`	Connection timeout	`10s`
`MONGODB_SERVER_SELECTION_TIMEOUT`	Server selection timeout	`5s`
`MONGODB_SOCKET_TIMEOUT`	Socket timeout	`30s`
`MONGODB_MAX_POOL_SIZE`	Maximum connection pool size	`100`
`MONGODB_MIN_POOL_SIZE`	Minimum connection pool size	`5`
`MONGODB_MAX_CONN_IDLE_TIME`	Max connection idle time	`5m`
`MONGODB_WRITE_CONCERN`	MongoDB write concern: `1` or `majority`	`majority`
`MONGODB_WRITE_JOURNAL`	Require journaled MongoDB writes	`true`

High Availability Configuration

Variable	Description	Default
`DISABLE_HIGH_AVAILABILITY`	Disable HA mode	`false`
`LOCK_TTL_SECONDS`	Leadership lock TTL	`30`
`LEADER_HEARTBEAT_INTERVAL`	Leader heartbeat frequency	`10s`
`LEADER_ELECTION_POLLING_INTERVAL`	Follower polling frequency	`5s`
`BLOCK_SYNC_INTERVAL`	Follower finalized-block synchronization interval	`1s`
`SERVER_ID`	Unique server identifier	`{hostname}-{pid}`
`LOCK_ID`	Unique lock identifier	`aggregator_leader_lock`

Logging Configuration

Variable	Description	Default
`LOG_LEVEL`	Logging level (debug, info, warn, error)	`info`
`LOG_FORMAT`	Log format (json, text)	`json`
`LOG_OUTPUT`	Log output (stdout, stderr, file path)	`stdout`
`LOG_ENABLE_JSON`	Enable JSON formatted logs	`true`
`LOG_ENABLE_ASYNC`	Enable asynchronous logging for better performance	`true`
`LOG_ASYNC_BUFFER_SIZE`	Buffer size for async logging	`10000`
`LOG_FILE_PATH`	Log file path; empty disables file logging and rotation	``
`LOG_MAX_SIZE_MB`	Rotate the log file once it reaches this size	`100`
`LOG_MAX_BACKUPS`	Rotated log files to retain	`30`
`LOG_MAX_AGE_DAYS`	Days to retain rotated log files	`30`
`LOG_COMPRESS_BACKUPS`	Compress rotated log files	`true`

Processing Configuration

Variable	Description	Default
`BATCH_LIMIT`	Batch-size hint; currently only logged at startup, not enforced	`1000`
`MAX_COMMITMENTS_PER_ROUND`	Cap on commitments collected per precollected round (child mode; standalone/`bft-shard` only when `USE_REDIS_FOR_COMMITMENTS=true`)	`20000`
`COLLECT_PHASE_DURATION`	Fixed collection window before proposing a round (non-child modes, non-precollected rounds)	`200ms`
`COLLECT_MINI_BATCH_SIZE`	SMT/proposal staging mini-batch size during collection	`500`
`COMMITMENT_STREAM_BUFFER_SIZE`	Buffer between the queue streamer and round collection	`50000`
`PRECOLLECTOR_GRACE_PERIOD`	Extra wait before cutting a precollected round snapshot	`0s`
`SKIP_DUPLICATE_CHECK`	Skip the finalized-record lookup on submit	`true`
`PARENT_COLLECT_PHASE_DURATION`	Collection window before proposing a round in `parent` mode	`200ms`

Storage Configuration

Variable	Description	Default
`USE_REDIS_FOR_COMMITMENTS`	Use Redis for commitment queue (instead of MongoDB)	`false`
`REDIS_HOST`	Redis server hostname (single-endpoint mode)	`localhost`
`REDIS_PORT`	Redis server port (single-endpoint mode)	`6379`
`REDIS_PASSWORD`	Redis server password (data nodes)	``
`REDIS_DB`	Redis database number	`0`
`REDIS_STREAM_NAME`	Redis stream name for commitments (allows multiple shards to share a Redis instance)	`commitments`
`REDIS_FLUSH_INTERVAL`	Interval for flushing pending commitments to Redis	`100ms`
`REDIS_MAX_BATCH_SIZE`	Maximum batch size before forcing flush	`5000`
`REDIS_ACK_BATCH_SIZE`	Commitments acknowledged per XACK batch	`1000`
`REDIS_DELETE_AFTER_ACK`	Delete stream entries once acknowledged	`true`
`REDIS_MAX_STREAM_LENGTH`	Stream length before trimming	`1000000`
`REDIS_CLEANUP_INTERVAL`	Interval between stream trim checks	`5m`
`REDIS_POOL_SIZE`	Connection pool size	`10`
`REDIS_MIN_IDLE_CONNS`	Minimum idle connections kept in the pool	`2`
`REDIS_MAX_RETRIES`	Retries per Redis command	`3`
`REDIS_DIAL_TIMEOUT`	Connection dial timeout	`5s`
`REDIS_READ_TIMEOUT`	Read timeout	`3s`
`REDIS_WRITE_TIMEOUT`	Write timeout	`3s`

SMT Backend

The default SMT backend is in-memory. A RocksDB-backed SMT can be enabled for bft-shard deployments (standalone or HA) by building with the rocksdb tag and setting SMT_BACKEND=rocksdb.

go build -tags rocksdb ./cmd/aggregator

Variable	Description	Default
`SMT_BACKEND`	SMT backend: `memory` or `rocksdb`	`memory`
`SMT_DISK_PATH`	RocksDB data directory when `SMT_BACKEND=rocksdb`	required
`SMT_NODE_KEY_FORMAT`	RocksDB node-key layout: `depth-major` or `prefix-major`	`depth-major`
`SMT_ROCKSDB_CACHE_MB`	RocksDB block cache size in MB	`1024`
`SMT_ROCKSDB_BG_JOBS`	RocksDB background jobs	`8`
`SMT_ROCKSDB_SUBCOMPACTIONS`	RocksDB subcompactions	`4`
`SMT_ROCKSDB_BLOOM_BITS`	Bloom filter bits per key	`10`
`SMT_ROCKSDB_MEMTABLE_MB`	RocksDB write buffer size in MB	`64`
`SMT_MATERIALIZE_WORKERS`	Parallel workers for SMT materialization	`16`
`SMT_PRECOMPUTE_PROOFS`	Precompute inclusion proof responses at finalization	`false`
`SMT_PROOF_METADATA_CACHE_ENTRIES`	Cached proof metadata entries	`250000`

RocksDB SMT with HA is supported only in bft-shard mode. It is rejected for application-level parent/child sharding modes; use SMT_BACKEND=memory there. Changing SMT_NODE_KEY_FORMAT requires a fresh or separately seeded SMT_DISK_PATH; an existing database opened with the wrong layout fails startup. When a non-empty RocksDB SMT is behind finalized MongoDB history, startup replays all missing finalized blocks in bounded pages. Replay has no block-count limit and may take significant time for a substantially stale node. Each block is committed independently, so interrupted replay resumes from the last committed RocksDB block.

Redis Sentinel (HA)

Set REDIS_SENTINEL_ADDRS to switch the client to Sentinel-backed failover. When set, REDIS_HOST/REDIS_PORT are ignored and REDIS_MASTER_NAME is required.

Variable	Description	Default
`REDIS_SENTINEL_ADDRS`	Comma-separated `host:port` list of Sentinel nodes. Empty = single-endpoint mode.	``
`REDIS_MASTER_NAME`	Sentinel master name to track. Required when `REDIS_SENTINEL_ADDRS` is set.	``
`REDIS_SENTINEL_PASSWORD`	Password for authenticating to Sentinel nodes.	``
`REDIS_SENTINEL_USERNAME`	ACL username for authenticating to Sentinel nodes.	``

Example:

USE_REDIS_FOR_COMMITMENTS=true \
REDIS_SENTINEL_ADDRS=sentinel-1:26379,sentinel-2:26379,sentinel-3:26379 \
REDIS_MASTER_NAME=mymaster \
REDIS_PASSWORD=secret \
make run

BFT Configuration

Variable	Description	Default
`BFT_ENABLED`	Enables or disables the BFT client integration.	`true`
`BFT_ADDRESS`	The libp2p multiaddress for the BFT client to listen on.	`/ip4/0.0.0.0/tcp/9000`
`BFT_RPC_ADDRESS`	The BFT node's RPC API address.	`http://127.0.0.1:8002`
`BFT_ANNOUNCE_ADDRESSES`	Comma-separated list of public callback multi-addresses to announce to other peers.	`""`
`BFT_BOOTSTRAP_ADDRESSES`	Comma-separated list of bootstrap peer addresses.	`""`
`BFT_BOOTSTRAP_CONNECT_RETRY`	Number of retries for connecting to bootstrap peers.	`3`
`BFT_BOOTSTRAP_CONNECT_RETRY_DELAY`	Delay between bootstrap connection retries (in seconds).	`5`
`BFT_HEARTBEAT_INTERVAL`	How often the BFT client checks for inactivity.	`1s`
`BFT_INACTIVITY_TIMEOUT`	Duration of inactivity before the BFT client sends a new handshake.	`5s`
`SIGNING_KEY_FILE`	Path to the aggregator's signing key file (`keys.json`); also supplies the BFT key conf.	`""`
`BFT_SHARD_CONF_FILE`	Path to the aggregator shard configuration file.	`bft-config/shard-conf-7_0.json`
`BFT_TRUST_BASE_FILES`	Comma-separated list of paths to trust base files.	`bft-config/trust-base.json`

API Endpoints

JSON-RPC 2.0 Methods

All JSON-RPC requests should be sent as POST to / with Content-Type: application/json.

`certification_request`

Submit a state transition request to the aggregation layer with cryptographic validation.

Request:

{
  "jsonrpc": "2.0",
  "method": "certification_request",
  "params": "<hex-encoded CertificationRequest CBOR>",
  "id": 1
}

Params is a hex-encoded api.CertificationRequest CBOR payload. The payload contains raw 32-byte values for stateId, sourceStateHash, and transactionHash; these fields do not include algorithm prefixes.

Which corresponds to Go data structures:

// CertificationRequest represents the certification_request JSON-RPC request,
// sometimes also referred to as StateTransitionCertificationRequest, Commitment or UnicityServiceRequest.
type CertificationRequest struct {
	_ struct{} `cbor:",toarray"`
	Version types.Version

	// StateID is the unique identifier of the certification request, used as a key in the state tree.
	// In v2 it is the raw 32-byte hash of the CBOR array
	// [CertificationData.OwnerPredicate, CertificationData.SourceStateHash].
	StateID StateID

	// CertificationData contains the necessary cryptographic data needed for the CertificationRequest.
	CertificationData CertificationData

	AggregateRequestCount uint64
}

// CertificationData represents the necessary cryptographic data needed for a state transition CertificationRequest.
type CertificationData struct {
	_ struct{} `cbor:",toarray"`
	// Version must be 2; any other value is rejected at decode.
	Version types.Version `json:"version"`

	// OwnerPredicate is the owner predicate in format: CBOR[engine: uint, code: byte[], params: byte[]].
	//
	// In case of standard PayToPublicKey predicate the values must be:
	//  - engine = 01 (plain CBOR uint value of 1)
	//  - code = 4101 (byte array of length 1 containing the CBOR encoding of uint value 1)
	//  - params = 5821 000102..20 (byte array of length 33 containing the raw bytes of the public key value)
	OwnerPredicate Predicate `json:"ownerPredicate"`

	// SourceStateHash is the raw 32-byte hash of the source data.
	SourceStateHash SourceStateHash `json:"sourceStateHash"`

	// TransactionHash is the raw 32-byte hash of the transaction data.
	// It commits to ExpiresAt, so changing the deadline invalidates the witness.
	TransactionHash TransactionHash `json:"transactionHash"`

	// ExpiresAt is the exclusive certification request deadline in Unix seconds,
	// or null when the requester left the deadline to the service. It occupies a
	// fixed position in the encoding either way. When absent, the service derives
	// a deadline from consensus time and DEFAULT_REQUEST_TTL.
	ExpiresAt *uint64 `json:"expiresAt"`

	// Witness is the "unlocking part" of owner predicate. In case of PayToPublicKey owner predicate the witness must be
	// a signature created on the hash of CBOR array[SourceStateHash, TransactionHash],
	// in Unicity's [R || S || V] format (65 bytes).
	Witness HexBytes `json:"witness"`
}

Response:

{
  "jsonrpc": "2.0",
  "result": {
    "status": "SUCCESS"
  },
  "id": 1
}

Validation Statuses:

SUCCESS - Certification request accepted and will be included in next block
INVALID_PUBLIC_KEY_FORMAT - Invalid secp256k1 public key
INVALID_SIGNATURE_FORMAT - Invalid signature format or length
SIGNATURE_VERIFICATION_FAILED - Witness doesn't verify against SHA256(CBOR[sourceStateHash, transactionHash]) and the predicate's public key
STATE_ID_MISMATCH - StateID doesn't match SHA256(CBOR[ownerPredicate, sourceStateHash])
INVALID_SOURCE_STATE_HASH_FORMAT - SourceStateHash is not exactly 32 bytes
INVALID_TRANSACTION_HASH_FORMAT - TransactionHash is not exactly 32 bytes
INVALID_SHARD - The certification request was sent to the wrong shard
REQUEST_EXPIRED - The round reference time has reached the request's exclusive deadline
SERVICE_NOT_READY - Consensus reference time is not yet available
STATE_ID_EXISTS - A record for this stateId was already finalized (returned only when SKIP_DUPLICATE_CHECK=false; the check is off by default)
UNKNOWN - The owner predicate is malformed (engine is not 1, or code is not the single byte 0x01)

`get_inclusion_proof.v2`

Retrieve the v2 inclusion proof for a submitted certification request.

The stateId must be exactly 64 hex characters (32 raw bytes).

Request:

{
  "jsonrpc": "2.0",
  "method": "get_inclusion_proof.v2",
  "params": {
    "stateId": "c7aa6962316c0eeb1469dc3d7793e39e140c005e6eea0e188dcc73035d765937"
  },
  "id": 2
}

Response:

{
  "jsonrpc": "2.0",
  "result": "<hex-encoded CBOR>",
  "id": 2
}

The result field is a hex-encoded CBOR array whose proof is tagged. Every inclusion proof carries the reference time at which its leaf was created, independently of request-deadline policy:

[blockNumber, #39033([1, certificationData, referenceTime, certificateBytes, unicityCertificate])]

certificationData is the certification data for inclusion proofs, or null for non-inclusion proofs.
referenceTime is the round time fixed when the leaf was created.
For inclusion proofs, certificateBytes is the binary inclusion certificate: bitmap[32] || sibling_1[32] || ... || sibling_n[32], where n = popcount(bitmap). Siblings are in root-to-leaf order. For non-inclusion proofs, certificateBytes is an exclusion certificate: k_l[32] || h_l[32] || bitmap[32] || siblings... (exclusion proof generation is not yet implemented).
The expected SMT root is always taken from UC.IR.h (input record hash of the Unicity Certificate). No root field appears in the certificate itself.

Hash rules (Yellowpaper-aligned):

Value: SHA-256(CBOR([transactionHash, referenceTime])) for every inclusion proof
Leaf: H(0x00 || key || value)
Inner node (two children): H(0x01 || depth_byte || region(key, depth) || left || right)
Inner node (one child): passthrough (child hash unchanged)

region(key, depth) is the 32-byte key prefix addressing the node: the first depth bits of the key with all lower-significance bits cleared. Omitting it verifies only for proofs with no siblings. See docs/inclusion-proof-wire.md.

Key encoding: 32 bytes, big-endian (MSB-first) bit addressing. bit(key, d) = (key[d/8] >> (7 - d%8)) & 1.

Verification pseudocode:

h = H(0x00 || key || value)
j = len(siblings)
for d in 255..=0:
    if bitmap bit d is not set: continue
    j -= 1
    r = region(key, d)          # 32 bytes: first d bits of key, rest cleared
    if bit(key, d) == 1:        # descent went right, sibling is the left child
        h = H(0x01 || d || r || siblings[j] || h)
    else:
        h = H(0x01 || d || r || h || siblings[j])
assert j == 0 and h == UC.IR.h

`get_block_height`

Retrieve the current blockchain height.

Request:

{
  "jsonrpc": "2.0",
  "method": "get_block_height",
  "params": {},
  "id": 3
}

Response:

{
  "jsonrpc": "2.0",
  "result": {
    "blockNumber": "123"
  },
  "id": 3
}

`get_block`

Retrieve detailed information about a specific block.

Request:

{
  "jsonrpc": "2.0",
  "method": "get_block",
  "params": {
    "blockNumber": "123"
  },
  "id": 4
}

Response:

{
  "jsonrpc": "2.0",
  "result": {
    "block": {
      "index": "123",
      "chainId": "unicity",
      "shardId": 0,
      "version": "1.0",
      "forkId": "testnet",
      "rootHash": "0000b67ebbbb3a8369f93981b9d8b510a7b8e72fc1e1b8a83b7c0d8a3c9f7e4d",
      "previousBlockHash": "0000a1b2c3d4e5f6789012345678901234567890123456789012345678901234",
      "noDeletionProofHash": "",
      "createdAt": "1734435600000",
      "unicityCertificate": "d903ef8701d903f08a01190146005844303030303936613239366432323466323835633637626565393363333066386133303931353766306461613335646335623837653431306237383633306130396366633758443030303039366132393664323234663238356336376265653933633330663861333039313537663064616133356463356238376534313062373836333061303963666337401a68553075f600f65820d4b5491031d8a9365555a01fa4d9805e32a4205c15fa19e53dc7f27ad4c534e058204296135d76b6345cdffaf57b434f6bd5c3579f3843731fab79e1e5a74a6091c982418080d903f683010780d903e9880103190737001a685530a658200b98a86c69c788bc54773d62cfd053ef54cf495bdb9a4b8298ad6c99966de7e058201d2b93c6e36694c316302b9cf9bf3c6ca076b085d6aaeb1d1874cd23301fa3f4a3783531365569753248416d326857486d66794a36484143696476367934686f377655323778504365436f5253515873694443595937654358417661160bc40a6a8722bd025ab49449dec2cee4a4680cc20f9f4fb2e1328c2f2e511a0390678a911b81a26d0171bfc43e813a01da7458c15558abb954bd2a52e501783531365569753248416d3665514d72327351566263575a73505062706332537537416e6e4d5647487043323350557a47544141546e7058410be3d9a494027aaed1d052145f8bd78ec5f909c1eeaa62e4a0aa79de1aef6108483aa8ff9253fb1d1c73407f49f428d246813780ed3648a92efa4c674fb5531401783531365569753248416d424a394c733865333662776b6a4c3574677737327a6b533578346479636a625a665956614e52676e7447317258415bd2c3b0ca0683c39e66129027eee216a66fc35eca1c58b5ba3e5a99dae4e97357893f88f7e91f70a16cccdfc7bfc9fa46757e2e1b1126bd5145af70a39e4bdb00"
    },
    "totalCommitments": "1"
  },
  "id": 4
}

`get_block_records`

Retrieve all certification requests included in a specific block.

Request:

{
  "jsonrpc": "2.0",
  "method": "get_block_records",
  "params": {
    "blockNumber": "123"
  },
  "id": 5
}

Response:

{
  "jsonrpc": "2.0",
  "result": {
    "aggregatorRecords": [
      {
        "stateId": "c7aa6962316c0eeb1469dc3d7793e39e140c005e6eea0e188dcc73035d765937",
        "certificationData": {
          "version": 2,
          "ownerPredicate": {
            "engine": 1,
            "code": "AQ==",
            "params": "AnxP34noE4s2A5enKFypm4Y0mdJvPBZSJR/PaA9NZIgs"
          },
          "sourceStateHash": "539cb40d7450fa842ac13f4ea50a17e56c5b1ee544257d46b6ec8bb48a63e647",
          "transactionHash": "c5f9a1f02e6475c599449250bb741b49bd8858afe8a42059ac1522bff47c6297",
          "expiresAt": 1755003600,
          "witness": "65ed0261e093aa2df02c0e8fb0aa46144e053ea705ce7053023745b3626c60550b2a5e90eacb93416df116af96872547608a31de1f8ef25dc5a79104e6b69c8d00"
        },
        "referenceTime": 1755000000,
        "aggregateRequestCount": "1",
        "blockNumber": "123",
        "leafIndex": "0",
        "createdAt": "1734435600000"
      }
    ]
  },
  "id": 5
}

`get_no_deletion_proof`

Retrieve the global no-deletion proof for the aggregator data structure. Not implemented yet — in standalone/child mode this returns a fixed placeholder proof, and in parent mode it returns an error.

Request:

{
  "jsonrpc": "2.0",
  "method": "get_no_deletion_proof",
  "params": {},
  "id": 6
}

Response:

{
  "jsonrpc": "2.0",
  "result": {
    "noDeletionProof": {
      "proof": "6d6f636b5f6e6f5f64656c6574696f6e5f70726f6f66",
      "createdAt": "1734435600000"
    }
  },
  "id": 6
}

HTTP Endpoints

`GET /health`

Returns the health status and role of the service.

Response:

{
  "status": "ok",
  "role": "leader",
  "serverId": "hostname-1234",
  "sharding": {
    "mode": "standalone",
    "shardIdLen": 4,
    "shardId": 0
  },
  "details": {
    "database": "connected",
    "commitment_queue": "42",
    "commitment_queue_status": "healthy"
  }
}

`PUT /api/v1/trustbases`

Adds trust base to the trust base store. The request body must be a valid trust base in json format.

Example curl request

curl -X PUT -H 'Content-Type: application/json' -d @./bft-config/trust-base.json http://localhost:3000/api/v1/trustbases

If trust base was stored successfully then status 200 with empty response body is returned:

{}

If trust base is invalid error then status 400 with error cause is returned:

{
  "error":"failed to store trust base: trust base already exist for epoch 1: trust base already exists"
}

`GET /docs`

Returns executable interactive HTML API documentation page with live testing capabilities (if ENABLE_DOCS=true).

The documentation includes:

🚀 Live API testing - Execute requests directly from the browser
📋 cURL export - Copy commands for terminal use
⌨️ Keyboard shortcuts - Ctrl+Enter to send requests
🎯 Status indicators - Response times and success/error status
🗑️ Clear - Reset the response panel for a method
📱 Responsive design - Works on desktop and mobile
💾 Real-time responses - JSON pretty-printed in a monospace response panel

Development

Building

# Build binary
make build

# Build and run
make run

# Run tests
make test

# Run with race detection
make test-race

# Format code
make fmt

# Lint code (requires golangci-lint)
make lint

# Clean build artifacts
make clean

Docker

# Clean rebuild (stops, removes data, rebuilds)
make docker-run-clean
# This command automatically uses current user's UID/GID on Linux/macOS

Project Structure

aggregator-go/
├── cmd/
│   ├── aggregator/         # Main application entry point
│   └── performance-test/   # Built-in performance testing tool
├── internal/               # Private application code
│   ├── config/            # Configuration management
│   ├── gateway/           # HTTP server and JSON-RPC handlers
│   ├── service/           # Business logic layer
│   ├── round/             # Round management and block creation
│   ├── signing/           # Cryptographic validation (secp256k1)
│   ├── smt/               # Sparse Merkle Tree implementation
│   ├── storage/           # Storage interfaces and implementations
│   │   ├── interfaces/    # Storage interface definitions
│   │   └── mongodb/       # MongoDB implementations
│   ├── models/            # Data models and types
│   └── logger/            # Logging utilities
└── pkg/                   # Public/reusable packages
    ├── api/              # Public API types
    └── jsonrpc/          # JSON-RPC server implementation

Database Collections

The service creates and manages the following MongoDB collections:

commitments - Temporary storage for pending commitments
aggregator_records - Finalized certification request records with proofs
blocks - Blockchain blocks with metadata
smt_nodes - Sparse Merkle Tree leaf nodes
trust_bases - Root trust base documents (BFT network trust base)
leadership - High availability leader election state

All collections include proper indexes for efficient querying.

Performance Testing

The service includes a built-in performance testing tool that generates cryptographically valid commitments:

# Run performance test (defaults to a single shard target at https://localhost:3001;
# set SHARD_TARGETS to point at your aggregator, e.g. SHARD_TARGETS="http://localhost:3000:1")
make performance-test

# Run performance test against a remote endpoint with authentication
SHARD_TARGETS="http://localhost:8080:1" AUTH_HEADER='Bearer supersecret' make performance-test

# Sharded performance test (provide shard targets with shardID suffix)
SHARD_TARGETS="http://localhost:3001:3,http://localhost:3002:2" TEST_DURATION=10s REQUESTS_PER_SEC=100 go run ./cmd/performance-test

Performance Test Features:

✅ Cryptographically Valid Data - Real secp256k1 key pairs and signatures
✅ Raw v2 Hash Format - 32-byte SHA256 state and transaction hashes
✅ Deterministic StateIDs - Calculated as SHA256(CBOR[ownerPredicate, sourceStateHash])
✅ High Concurrency - Configurable worker count and request rate
✅ Block Monitoring - Tracks certification requests per block and throughput
✅ Real-time Metrics - Success rate, failure rate, and RPS tracking

Sample Output:

Starting aggregator performance test...
Sharding mode: app
Targets (1 shards):
  - shard-7 (https://localhost:3001) shardMask=7
Duration: 30s
Submission workers: 20
Proof scheduling: exact per-submission timer (PROOF_WORKERS ignored, value=10)
Proof initial delay: 2.5s
Proof retry delay: 1s
Server proof-readiness metric: aggregator_proof_readiness_seconds_bucket (direct /metrics scrape)
HTTP client pool size: 4
H2C: enabled (HTTP/2 cleartext for plain HTTP)
Target RPS: 2000
----------------------------------------
Testing connectivity to https://localhost:3001...
✓ Connected successfully to https://localhost:3001
✓ Starting block number for https://localhost:3001: 42
...

High Availability

The service implements a MongoDB-based leader election system:

Distributed Processing: All servers handle API requests
Leader Election: Only one server creates blocks and manages consensus
Automatic Failover: Leader election with configurable TTL
Health Monitoring: /health endpoint reports current role

Leadership Roles

leader - Actively creating blocks and managing consensus
follower - Processing API requests, monitoring for leadership
standalone - Single server mode (HA disabled)
parent-leader - Parent aggregator actively aggregating child shard roots
parent-follower - Parent aggregator processing API requests, monitoring for leadership
parent-standalone - Single parent aggregator (HA disabled)

Sharding

The aggregator supports two orthogonal sharding strategies, selected by SHARDING_MODE:

Application-level sharding (parent / child) — aggregator-layer split: one parent aggregator aggregates the SMTs of multiple children. Described in the rest of this section.
BFT-side sharding (bft-shard) — BFT-layer split: multiple aggregators act as shard validators of a single multi-shard BFT partition, and shard-inclusion is proved directly by the ShardTreeCertificate embedded in the UnicityCertificate. Described in BFT-side sharding.

The two modes share the same routing input (the raw 32-byte stateId, read MSB-first) but use different shard-ID semantics; they are not interchangeable.

Application-level sharding (`SHARDING_MODE=parent` / `child`)

To support horizontal scaling, the aggregators can be run in a sharded configuration consisting of one parent aggregator and multiple child aggregators. In this mode, the global Sparse Merkle Tree (SMT) is split across the child nodes, and agents must submit their certification requests to the correct child node.

For a more detailed technical explanation of the sharded SMT structure, please refer to the official specification: https://github.com/unicitynetwork/specs/blob/main/smt.md

Certification Request Routing

The requests are assigned to a shard based on the most significant (leading) bits of their state identifier. The number of bits used to determine the shard is defined by the SHARD_ID_LENGTH configuration.

For example SHARD_ID_LENGTH: 1 means that the leftmost 1 bits of state identifier determines the correct shard. In this case there would be 2 shards e.g. certification requests starting with bit 0 would go to the shard whose shardID is 0b10, and certification requests starting with bit 1 would go to the shard whose shardID is 0b11.

In sharded setup only the parent aggregator talks to the BFT node.

Shard ID Encoding

The shardID is a unique identifier for each shard that includes a 1 as its most significant bit (MSB). This prefix bit ensures that the leading zeros are preserved for bit manipulations.

Examples

For SHARD_ID_LENGTH: 1 the valid shardIDs are 0b10 (2) and 0b11 (3), for a total of two shards.
For SHARD_ID_LENGTH: 2 the valid shardIDs are 0b100 (4), 0b101 (5), 0b110 (6) and 0b111 (7), for a total of four shards.

A child aggregator validates incoming certification requests to ensure they belong to its shard. If a certification request is sent to a wrong shard, the aggregator will reject it.

Example Sharded Setup

The following diagram illustrates a sharded setup with one parent and two child aggregators for SHARD_ID_LENGTH: 1.

          +--------------------+
          | Parent Aggregator  |
          | (2-leaf SMT)       |
          +--------------------+
           /                  \
          /                    \
+----------------+     +----------------+
| Child Agg. #1  |     | Child Agg. #2  |
| ShardID = 0b10 |     | ShardID = 0b11 |
| (handles 0...) |     | (handles 1...) |
+----------------+     +----------------+
        ^                      ^
        |                      |
+----------------+     +----------------+
| Agent sends    |     | Agent sends    |
| commitment     |     | commitment     |
| ID = 0xxx...   |     | ID = 1xxx...   |
+----------------+     +----------------+

Configuration

The sharded setup is configured via environment variables, as seen in sharding-compose.yml.

A parent aggregator is configured with:

environment:
  SHARDING_MODE: "parent"
  SHARD_ID_LENGTH: 1

A child aggregator is configured with its unique shardID and the address of the parent, for example:

Shard-1:

environment:
  SHARDING_MODE: "child"
  SHARDING_CHILD_SHARD_ID: 3 # (binary 0b11)
  SHARDING_CHILD_PARENT_RPC_ADDR: http://aggregator-root:3000

Shard-2:

environment:
  SHARDING_MODE: "child"
  SHARDING_CHILD_SHARD_ID: 2 # (binary 0b10)
  SHARDING_CHILD_PARENT_RPC_ADDR: http://aggregator-root:3000
  SHARDING_CHILD_PARENT_POLL_INTERVAL: 100ms   # default
  SHARDING_CHILD_PARENT_POLL_TIMEOUT: 5s       # default

BFT-side sharding (`SHARDING_MODE=bft-shard`)

In bft-shard mode, multiple aggregators are deployed as shard validators of a single multi-shard BFT partition. Each aggregator owns one shard, talks directly to the BFT rootchain, and the UnicityCertificate it receives embeds a ShardTreeCertificate that binds its local SMT root into the partition root. There is no parent aggregator — shard-inclusion is proved by the embedded certificate rather than by a per-round polling loop.

This mode is orthogonal to parent/child: it uses the same routing key (the raw 32-byte stateId, read MSB-first) but a different shard-ID encoding (bit-strings instead of sentinel-prefixed integers) and a different admission rule.

Routing semantics

Routing key: the raw 32-byte stateId.
Shard identifiers are bit-strings, e.g. "0", "1", "101". The first N bits of stateId (bit 7 of byte 0 first, descending to bit 0 of byte 31) must equal the shard's bit-string. This matches types.ShardID.Comparator() in bft-go-base.
A request whose stateId does not match the local shard's prefix is rejected with INVALID_SHARD.
Shard prefixes must form a prefix-free covering set: every possible stateId maps to exactly one shard.

Example: 1-bit split

                        +---------------------+
                        |  BFT rootchain      |
                        |  (partition id 7)   |
                        +---------------------+
                         /                   \
                        /                     \
+---------------------------+    +---------------------------+
| aggregator-bft-shard0     |    | aggregator-bft-shard1     |
| SHARDING_MODE=bft-shard   |    | SHARDING_MODE=bft-shard   |
| shard-id = 0x40 ("0")     |    | shard-id = 0xC0 ("1")     |
| accepts stateId with MSB=0|    | accepts stateId with MSB=1|
| listens on :3001          |    | listens on :3002          |
+---------------------------+    +---------------------------+

Note on shard-conf hex values: 0x40 encodes the bit-string "0" under the trailing-1 end-marker convention in bft-go-base/types/bitstring.go; 0x80 is the length-0 empty shard. The two children of the empty-shard root split are 0x40 and 0xC0.

Configuration

A bft-shard aggregator is configured with:

environment:
  SHARDING_MODE: "bft-shard"
  BFT_ENABLED: "true"
  BFT_SHARD_CONF_FILE: "/app/bft-config/shard_0/shard-conf-7_0.json"
  BFT_TRUST_BASE_FILES: "/app/bft-config/trust-base.json"
  BFT_RPC_ADDRESS: "http://bft-root:8002"
  # plus the standard BFT_ADDRESS / BFT_BOOTSTRAP_ADDRESSES / BFT_ANNOUNCE_ADDRESSES

Shard identity (partition ID, shard ID, genesis epoch) is loaded from BFT_SHARD_CONF_FILE, produced by ubft shard-conf generate ... --shard-id 0x40 .... All shard confs in the partition must be pre-provisioned to the rootchain; the aggregator does not register dynamically.

A bft-shard config must have a non-empty shard ID (length ≥ 1 bit). For a single-shard deployment, use SHARDING_MODE=standalone.

Quickstart (local Docker)

# Spin up 1 BFT rootchain + 2 shard aggregators + MongoDB + Redis.
make docker-run-bft-sh-clean

# Tail shard logs:
docker logs -f aggregator-bft-shard0   # serves on :3001 (shard "0")
docker logs -f aggregator-bft-shard1   # serves on :3002 (shard "1")

# Teardown:
docker compose -f bft-sharding-compose.yml down --remove-orphans

Other make targets:

make docker-run-bft-sh-clean-keep-tb — preserves BFT genesis/trust-base across restarts; reinitializes MongoDB/Redis only.
make docker-restart-bft-sh — rebuilds and restarts only the aggregators, leaving BFT nodes running.

Quickstart with local HA replicas

Use the standalone HA compose file for a local multi-replica bft-shard deployment with RocksDB SMT:

make docker-run-bft-sh-ha-clean

# Stable shard endpoints for SDK/perf tests:
#   shard "0": http://localhost:3001  (HAProxy -> active shard 0 replica)
#   shard "1": http://localhost:3002  (HAProxy -> active shard 1 replica)

# Tail replica logs:
docker logs -f aggregator-bft-shard0
docker logs -f aggregator-bft-shard0-b
docker logs -f aggregator-bft-shard1
docker logs -f aggregator-bft-shard1-b

# Teardown:
docker compose -f bft-sharding-ha-compose.yml down --remove-orphans

The base BFT-shard compose file remains the default single-replica topology. The HA compose file is standalone: it runs two replicas per shard, each with its own RocksDB directory, behind per-shard HAProxy instances that route to the current leader via /health/leader.

Driving test traffic

cmd/commitment submits individual v2 certification requests to a chosen endpoint. cmd/performance-test with SHARDING_MODE=bft-shard and SHARD_TARGETS of the form http://localhost:3001:0,http://localhost:3002:1 drives both shards in parallel and generates stateIds whose leading bits match each shard's prefix. Cross-shard traffic is rejected with INVALID_SHARD.

Client-side proof verification

Returned inclusion proofs should be verified on the client side. The Go API exposes the strict verifier as pkg/api.InclusionProofV2.Verify(req, vctx), which checks the proof against the trust base and expected shard/partition context.

Observability

/health reports the bft-shard identity via bftShardId (MSB-first bit-string; empty in non-bft-shard modes). The integer shardIdLen / shardId fields are zero in bft-shard mode.
Wrong-shard submissions return INVALID_SHARD at admission; the rejecting aggregator's log records the mismatch.

Error Handling

The service implements comprehensive JSON-RPC 2.0 error codes:

Code	Description
`-32700`	Parse error
`-32600`	Invalid request
`-32601`	Method not found
`-32602`	Invalid params
`-32603`	Internal error
`-32000`	Validation error
`-32001`	Commitment already exists
`-32002`	Commitment not found
`-32003`	Block not found
`-32004`	Database error
`-32005`	Consensus error
`-32006`	Concurrency limit exceeded

Performance Considerations

Connection Pooling: Configurable MongoDB connection pool
Concurrency Limiting: Prevents resource exhaustion
Request Correlation: Efficient logging with request IDs
Graceful Shutdown: Proper cleanup on termination
Batch Operations: Efficient database operations (when available)
Asynchronous Logging: Non-blocking log writes with configurable buffer size (enabled by default)
- Batched log processing reduces I/O overhead
- 10ms flush interval ensures timely log delivery
- Graceful shutdown flushes all pending logs

Cryptographic Implementation

Signature Validation

The service implements complete secp256k1 signature validation:

✅ Public Key Validation - Compressed 33-byte secp256k1 public keys
✅ Signature Verification - 65-byte signatures (64 bytes + recovery byte)
✅ StateID Validation - Deterministic calculation over owner predicate and source state hash
✅ Raw Hash Support - 32-byte source state and transaction hashes
✅ Transaction Signing - Signatures verified against SHA256(CBOR array [sourceStateHash, transactionHash])

Supported Algorithms

secp256k1 - Full implementation with btcec library
SHA256 - Hash algorithm for all cryptographic operations
Raw v2 Hashes - 32-byte SHA256 hashes without per-field algorithm prefixes

Validation Process

Owner Predicate Check - Verify the pay-to-public-key predicate (engine 1, code 0x01) and extract the public key from its params
Public Key Format - Validate compressed secp256k1 public key (33 bytes)
State Hash Format - Validate raw 32-byte source state hash
StateID Verification - Ensure StateID matches the owner predicate and source state hash
Signature Format - Validate 65-byte signature length
Transaction Hash Format - Validate raw 32-byte transaction hash
Signature Verification - Cryptographically verify the signature against SHA256(CBOR array [sourceStateHash, transactionHash])

Architecture Notes

Round Management

Round-based processing - Automated block creation
Batch processing - Multiple commitments per block
Leader-only block creation - High availability with single leader
Graceful shutdown - Proper cleanup of pending rounds

SMT Integration

Spec compliant - Exact hash matching with Unicity SMT specification
Memory efficient - Optimized for large-scale operations
Concurrent safe - Thread-safe operations with proper locking

Limitations

Submission Receipts: certification_request returns only {"status": ...} — there is no submission receipt object, signed or unsigned.

Contributing

Follow Go best practices and conventions
Write tests for new functionality
Update documentation concurrently with code changes
Use the provided Makefile for builds and testing

License

[License information to be added based on project requirements]

Support

For issues and questions:

Create GitHub issues for bugs and feature requests
Check the /health endpoint for service status
Review logs for detailed error information