Crabka

Crabka is a Rust implementation of Apache Kafka infrastructure. It speaks the Kafka wire protocol, stores records in Kafka-compatible log segments, and runs metadata on KRaft. The test suite runs Crabka against the official JVM clients and command-line tools.

Use Crabka when you want Kafka-compatible streaming infrastructure without a JVM runtime. Crabka gives you memory-safe Rust, async I/O, no ZooKeeper mode, and no GC pauses.

This repository is closing

Crabka was one monorepo. It is now sixteen repositories in the krabka-io organization, and each crate is renamed crabka-* to krabka-*.

Every component has a home. The extracted repositories are ahead of this one. They carry restructured modules, more tests, and crates that do not exist here. Read and change each component in its home repository. Do not send changes here.

This repository will be archived. After that it is read-only and kept only for history.

Component	Home repository
Broker, KRaft, log, tiered storage, benchmarks, verification	krabka-broker
Kafka wire protocol, metadata records, the protocol generator	krabka-protocol
Rust producer, consumer, and admin clients	krabka-client-rs
Streams clients	krabka-streams-rs, krabka-streams-java, krabka-streams-go
Connect runtime, Postgres CDC, replication	krabka-connect
Schema Registry	krabka-schema-registry
Metrics, traces, profiles, and logs	krabka-o11y, krabka-o11y-demo
Postgres-compatible engine (Gres)	gres
gRPC gateway, the four application SDKs, CloudEvents demo	krabka-gateway
CLI and the admin UI	krabka-cli
Kubernetes operator and the CRDs	krabka-operator
Partition rebalancer	krabka-rebalancer
Website and the WASM playground	krabka-io.github.io
Release tooling, base-image inputs, chart signing	tooling

The Kubernetes API group changed from crabka.io to krabka.io with the operator. The Helm chart signing key changed too: the old Crabka Charts <charts@crabka.dev> key is revoked, and charts are signed with Krabka Charts <charts@krabka.dev>. Signatures made with the old key do not verify.

The moves are recorded in the Extraction to krabka-io milestone.

Project Status

Crabka is beta, pre-1.0 software. The workspace version is in Cargo.toml.

The 0.4.0 milestone ships metadata downgrade and client rebootstrap, durable transaction recovery, diskless WAL failover, tiered offset reads, CloudEvents and queue semantics in the gateway, operator lifecycle work, and shared conformance coverage for five application SDKs. The finite outcomes for this milestone are closed; UNFINISHED_WORK.md records the remaining directional horizons and known limits.

The project is still greenfield infrastructure. There are no production users, and Crabka does not promise on-disk compatibility across versions yet. Use Crabka for evaluation, development, interoperability tests, and non-critical workloads while the project hardens.

Kafka compatibility is the primary constraint. The repository validates protocol encoding, record formats, storage behavior, KRaft metadata, and JVM tool interoperability against Apache Kafka behavior. This applies where those surfaces are in scope.

Why Crabka

• Kafka wire compatibility: the build generates the protocol codecs from Apache Kafka message schemas and checks them byte-for-byte against kafka-clients.
• JVM tooling works: acceptance tests drive tools such as kafka-topics.sh, kafka-configs.sh, kafka-acls.sh, kafka-consumer-groups.sh, kafka-leader-election.sh, and kafka-reassign-partitions.sh against Crabka.
• Rust runtime: Crabka uses tokio, forbids unsafe code across the workspace, and avoids JVM heap tuning and garbage-collection behavior.
• KRaft-native: Crabka stores metadata in a native KRaft quorum. ZooKeeper mode and ZooKeeper-to-KRaft migration are out of scope.
• Operations included: a Kubernetes operator, Prometheus metrics, OTLP tracing, Helm charts, OCI images, and a Cruise-Control-style partition rebalancer.
• Rust clients included: producer, consumer, admin, streams, schema-serde, gateway, connector, and replication crates.

Compatibility

Crabka targets Kafka's wire, storage, and operational semantics. JVM implementation internals are not compatibility goals.

Area	Status
Wire protocol and API version negotiation	Implemented
Kafka-compatible record batches, compression, and log segments	Implemented
KRaft metadata quorum and controller records	Implemented
Replication, ISR maintenance, leader election, and reassignment	Implemented
Idempotent and transactional produce / consume	Implemented
Classic and next-generation consumer groups	Implemented
Share groups / queues	Implemented
Tiered storage	Implemented, including Kafka 4.0 JVM segment-layout and producer-snapshot validation
TLS, SASL, delegation tokens, ACLs, and quotas	Implemented
Schema Registry-compatible REST service	Implemented
Kubernetes operator	Implemented, including Ingress and OpenShift Route listeners
Rust Streams client	Partial versus the full JVM Kafka Streams library
Kafka Connect-equivalent runtime	Partial; managed Postgres CDC workers, durable offsets, connector SPI, and KafkaConnector CRD are implemented
ZooKeeper mode and ZooKeeper-to-KRaft migration	Out of scope

For the detailed per-KIP breakdown, see docs/KIP_MATRIX.md.

For managed Postgres CDC setup, see docs/connect.md.

Install

Crabka is a Rust workspace. The pinned toolchain is in rust-toolchain.toml.

git clone https://github.com/robot-head/crabka.git
cd crabka
cargo build --workspace

Install the local broker and CLI binaries from a checkout:

cargo install --path crates/cli
cargo install --path crates/broker

The Rust client crates are published independently. They are maintained in krabka-client-rs, which is ahead of the copies under crates/; use that repository for client work.

The project publishes container images to GHCR and Docker Hub:

docker pull ghcr.io/robot-head/crabka-broker:latest
docker pull mirror.gcr.io/robothead/crabka-broker:latest

packaging/README.md gives the image build, signature, SBOM, and attestation details. charts/README.md documents the Helm chart usage.

Quick Start

Start a single local broker from the source tree:

export CRABKA_CLUSTER_ID=00000000-0000-0000-0000-000000000001
rm -rf target/crabka-data

cargo run -p crabka-cli --bin crabka -- format \
  --log-dir target/crabka-data \
  --cluster-id "$CRABKA_CLUSTER_ID" \
  --standalone \
  --node-id 1 \
  --controller-listener 127.0.0.1:9093

cargo run -p crabka-broker --bin crabka-broker -- \
  --log-dir target/crabka-data \
  --cluster-id "$CRABKA_CLUSTER_ID" \
  --broker-id 1 \
  --listen-addr 127.0.0.1:9092

In another shell, use normal Kafka tooling against the broker:

kafka-topics.sh \
  --bootstrap-server 127.0.0.1:9092 \
  --create \
  --topic demo \
  --partitions 1 \
  --replication-factor 1

kafka-console-producer.sh \
  --bootstrap-server 127.0.0.1:9092 \
  --topic demo

kafka-console-consumer.sh \
  --bootstrap-server 127.0.0.1:9092 \
  --topic demo \
  --from-beginning

crabka format initializes an empty log directory. To start again locally, stop the broker and delete target/crabka-data.

Documentation

• KIP implementation matrix
• Contributing guide
• Container image docs
• Helm chart docs
• Benchmark harness
• Style guides
• docs.rs package documentation
• Project website
• krabka-io organization, which owns the extracted components

Workspace

Crabka is a Cargo workspace. The main runtime path is below. The Maintained in column gives the repository that now owns each layer; the crate links point at the copies in this repository, which are frozen for extracted components.

flowchart LR
    clients[Kafka and Crabka clients] --> broker[crabka-broker]
    broker --> log[Kafka-compatible log]
    broker --> kraft[KRaft metadata quorum]
    broker --> remote[Tiered storage]
    broker --> telemetry[Metrics / logs / traces]
    operator[crabka-operator] --> broker
    registry[crabka-schema-registry] --> broker
    gateway[crabka-grpc-gateway] --> broker
    rebalancer[crabka-rebalancer] --> broker
    replicator[crabka-replicator] --> broker

Layer	Key crates	Maintained in
Broker runtime	crabka-broker, crabka-authz, crabka-security, crabka-telemetry	krabka-broker, krabka-protocol
CLI	crabka-cli	krabka-cli
Protocol, records, and storage	crabka-protocol, crabka-log, crabka-raft, crabka-metadata, crabka-remote-storage	krabka-protocol, krabka-broker
Rust clients	crabka-client-core, crabka-client-producer, crabka-client-consumer, crabka-client-admin, crabka-client-streams	krabka-client-rs, krabka-streams-rs
Services and integration	crabka-schema-registry, crabka-connect, crabka-connect-postgres, crabka-replicator	krabka-schema-registry, krabka-connect
Gateway and application SDKs	crabka-grpc-gateway, crabka-app-sdk, sdks/	this repository
Operations	crabka-operator, crabka-rebalancer	krabka-operator, krabka-rebalancer
Observability	crabka-blockstore, crabka-metrics, crabka-observability	krabka-o11y
Postgres-compatible engine (Chapter Gres)	crabka-gres, crabka-gres-control, crabka-gres-balancer, crabka-pgexec, crabka-pgwire, crabka-pgtypes, crabka-pgparser, crabka-pgkv, crabka-pgmvcc, crabka-pgcatalog, crabka-gres-fdw	gres
Tooling and harnesses	crabka-protocol-codegen, crabka-docgen, crabka-bench-driver, crabka-admin-ui, crabka-playground	this repository

Crate READMEs and rustdoc contain API-level usage details. For an extracted component, read them in its home repository rather than here.

Development

Prerequisites:

• Rust toolchain from rust-toolchain.toml
• JDK 17 for JVM differential tests
• Docker or a compatible container runtime for integration tests that use Kafka containers

Common checks:

cargo build --workspace
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace

Run JVM-backed differential and acceptance tests:

(cd tools/oracle && ./gradlew installDist)
cargo test --workspace -- --include-ignored

Regenerate the protocol code after you edit the Kafka schemas:

./tools/regenerate.sh
git diff crates/protocol/generated

CONTRIBUTING.md gives more contributor workflow details.

Roadmap

The near-term roadmap for this repository is the extraction itself: finishing the moves in the Extraction to krabka-io milestone, and finding a home for the gateway, SDK, tooling, and infrastructure work that has none yet.

Per-component roadmaps live in the repositories that own them. Work that continues here:

• The gRPC gateway and its four application SDKs, with shared conformance coverage.
• The protocol code generator that tracks upstream Kafka message schemas.
• The benchmark harnesses and the cross-crate integration suite.

docs/KIP_MATRIX.md and the design notes under docs/superpowers/specs give the detailed implementation status for this repository. They are not updated for changes made in the extracted repositories.

Contributing

This repository no longer takes changes. Find the component in the table under This repository is closing, then open the issue or pull request in its home repository under krabka-io. A change landed here does not reach that repository.

For the work still maintained here, start with CONTRIBUTING.md. Open an issue for a large design or compatibility change. Keep Kafka wire and behavior compatibility as the primary constraint.

Run cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, and the relevant tests before you open a pull request. release-plz uses conventional commits for automated versioning and changelog generation.

Security

Crabka includes authentication, authorization, TLS, mTLS, delegation-token, and OPA integration work, but it is still beta infrastructure. Do not use it as the sole security boundary for critical production systems yet.

If you find a security vulnerability, do not post exploit details in a public issue. Use GitHub private vulnerability reporting if the repository has it enabled. If not, contact the maintainers privately through the repository owner.

License

Crabka is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE.

Acknowledgements

Crabka is a derivative, compatibility-focused implementation of Apache Kafka protocols, record formats, and operational semantics. The project depends on the Apache Kafka schema corpus and JVM client/tool behavior as its compatibility oracle.