Sandbox backends: kfd, kql, k7d

k7 turns every sandbox into a Kubernetes Deployment whose pod runs inside a hardware-isolated microVM. How that VM is built, stored, snapshotted, and forked is the backend's job. Three backends exist today; a node can install any combination (k7 install --backend kfd,kql,k7d) and each sandbox picks one (k7 create --backend …, annotation k7.katakate.org/backend).

	kfd — kata-firecracker-devmapper	kql — kata-qemu-longhorn	k7d
VMM	Firecracker via Kata Containers	QEMU via Kata Containers	k7d, Katakate's purpose-built KVM VMM
containerd RuntimeClass	kata	kata-qemu	k7
Rootfs / storage	devmapper LVM thin-pool (needs a spare raw disk)	overlayfs + Longhorn PVC mounted at /mnt/state	cached erofs images on virtio-blk, guest tmpfs upper, reflink-XFS volume images
Named snapshots (k7 snapshot)	✗	✅ Longhorn VolumeSnapshot (disk-only, crash-consistent)	✗ by design — see below
k7 fork	✗	disk clone: snapshot → new PVC → cold boot	warm fork: CoW disk and memory of the live VM
k7 pause / resume	scale to 0/1 (state lost)	scale to 0/1, disk persists on the PVC	VM frozen in place: vCPUs stop, memory stays
k7 restore	✗	✅ boot a new sandbox from any named snapshot	✗ (fork the live sandbox instead)
Docker-in-VM sidecar	✅ (emptyDir data)	✅ (PVC-persistent data)	✅ (VM-lifetime data)

Measured numbers for all of this live in PERFORMANCE.md: on the same node, k7d creates in ~2.1s vs kql's ~17.1s, forks to a usable sandbox in ~2.4s vs ~46.7s (and the k7d fork inherits the source's memory), and pauses/resumes in ~0.2s/~0.3s vs ~1.3s/~4.1s — while kql keeps named persistent snapshots and the faster in-sidecar docker pull.

Which backend when?

• kfd — smallest attack surface and fast boots, when you don't need any snapshot/fork lifecycle. Requires a dedicated raw disk for the thin-pool.
• kql — durable sandboxes. The root disk is a replicated Longhorn volume: it survives pod restarts and node reboots, can be snapshotted by name, restored into brand-new sandboxes, and forked (at the cost of a full VM boot on the cloned disk).
• k7d — ephemeral-but-forkable compute, e.g. agent/RL rollouts. The whole VM (filesystem and RAM: running processes, page cache, sockets) can be forked in seconds, and pause/resume is instantaneous because nothing is torn down. State does not survive pod deletion — persistence is "fork it again", not "write it to a disk".

The k7d backend

Architecture: Linux amd64 / x86_64 only (Debian amd64 ≡ tarball x86_64). Unlike kfd / kql, there is no arm64 build yet.

What k7 installs (k7 install --backend k7d)

The Ansible playbook:

1. checks /dev/kvm exists (loud failure otherwise) and loads vhost_vsock + tun;
2. installs erofs-utils, xfsprogs, virtiofsd, and provisions a sparse reflink-capable XFS image loop-mounted at /var/lib/k7d/disks (warm forks clone writable volume images with FICLONE reflinks);
3. downloads the k7d release tarball (k7d_artifact_url, default the public Katakate/k7d GitHub release for k7d_version, currently 0.2.1) and runs the bundled install.sh, which installs k7d + containerd-shim-k7-v1 into /usr/local/bin, guest kernel/initramfs into /usr/local/share/k7d, and starts k7d.service (control socket /run/k7d/k7d.sock). Override with k7 install --k7d-version <ver> (same GitHub URL, other tag) or --k7d-artifact <path> / k7d_artifact_local_path (local tarball). The PPA k7 0.2.1 playbook still defaults to k7d 0.1.0 — pass --k7d-version 0.2.1 on that package until the next k7 cut.
4. registers the k7 runtime in the k3s containerd template with pod_annotations = ["k7d.katakate.org/*"] and without a BinaryName option (containerd resolves the shim from runtime_type = "io.containerd.k7.v1"), restarts k3s, and creates RuntimeClass k7;
5. labels the node k7.katakate.org/backend-k7d=true.

How a sandbox maps to a VM

One pod = one k7d microVM. The pod's containers (sandbox + optional docker sidecar) all run as runc containers inside that single VM — multi-container pods share the guest kernel, network namespace, and tmpfs. VM size follows the pod's CPU/memory limits (the shim reads the CRI sandbox annotations, so k7 create --cpu 2 --memory 1Gi gives a 2-vCPU/1 GiB VM with a host-side CFS cap).

The fork story

k7 fork src dst on a k7d sandbox does not copy a disk. Instead the new Deployment's pod carries two annotations:

k7d.katakate.org/fork-source-cluster: <source CRI sandbox id>
k7d.katakate.org/fork-source-vm:      <source CRI sandbox id>

The pod's containerd shim resolves the source VM through the k7d daemon and issues a fork_vm: the daemon briefly pauses the source, captures dirty pages, and builds the child from a MAP_PRIVATE CoW mapping of the source's memory plus reflink clones of its disk overlays. The child inherits everything — files, tmpfs, running processes, page cache — and diverges independently from that point. The source keeps running (its vsock identity is preserved). The fork pod then adopts the workload container already running inside the forked guest, so kubectl exec / k7 exec transparently target the inherited state.

Properties and limits (fail loudly, never silently):

• Warm fork works for single-workload sandboxes; forking a sandbox with a sidecar is rejected (no reliable container mapping — use kql for that).
• The fork is pinned to the source's node (the k7d daemon is node-local; cross-node fork is k7d's M12 roadmap item).
• If the fork pod is ever restarted by Kubernetes it re-forks from the (then current) source — a fork is a live branch, not a stored artifact.
• k7 snapshot/k7 restore are rejected on k7d: named, storable snapshots are a Longhorn/kql feature. k7d has its own richer VM snapshot trees (fork/rollback/suspend of whole VM states, including multi-VM clusters) driven through the k7d daemon API — see the k7d project. Whole-cluster fork (forking an inner k8s cluster of VMs as one unit) is deliberately a k7d-level feature, not a k7 verb.

k7d VM ops work on any node (per-node k7-agent)

k7d pause/resume/fork need three things that only exist on the node hosting the sandbox: the k7d daemon socket (/run/k7d/k7d.sock), the k3s containerd socket, and crictl. Since spec 18g this is handled by the k7-agent DaemonSet (kube-system, same k7-api:local image running k7.api.agent:app), so VM ops work through the API for a sandbox on any node:

• A VM op on a sandbox co-located with the k7-api pod runs directly (the deployment mounts both sockets and ships crictl).
• A VM op on a sandbox on any OTHER node is forwarded to the k7-agent pod on that node (POST /agent/v1/vm/{pause,resume,fork,lookup} on the pod IP). Forwarding authenticates with the shared token the install playbook writes to /etc/k7/agent_token (root, 0600) on every node; a CiliumNetworkPolicy limits pod-originated agent ingress to the k7-api pod. No Ready agent on the node / missing token → loud error, never a silent no-op.
• CLI on a node also works for any sandbox: local sandboxes talk to the local daemon, remote ones are forwarded the same way (root can read the token).
• A fork still lands on the source's node — the agent proxies to the node-local daemon; the cross-node fork data path is k7d spec 9a M12 (daemon side, not built yet). When it lands, only the forwarding target changes.

kql pause/resume/fork have none of these constraints (they are pure Kubernetes/Longhorn operations) and work through the API for any node.

Pause / resume

k7 pause on k7d asks the daemon (over /run/k7d/k7d.sock) to stop the VM's vCPU threads and park its device workers: RAM, devices, and the vsock CID all stay. The pod object remains scheduled (annotated k7.katakate.org/k7d-paused=true), so k7 resume is just "restart the vCPU loop" — sub-second, and every in-memory byte survives. Compare kql, where pause scales the Deployment to zero (the VM is destroyed; only the Longhorn disk survives) and resume pays a full VM boot.

Egress modes

A sandbox has exactly one of three egress modes (spec 18f issue 4):

Mode	CLI	YAML / API	Result
block-all (CLI default)	k7 create … (no egress flag)	egress_whitelist: []	deny-all egress NetworkPolicy
whitelist	--egress <entry> (repeatable)	egress_whitelist: [<entries>]	only listed CIDRs/domains
open	--egress-open	omit egress_whitelist (or null)	no egress policy at all

--egress-open and --egress are mutually exclusive. Note the asymmetric defaults, kept for backward compatibility: the CLI without flags is block-all, while an API/YAML request that omits egress_whitelist entirely is open.

Wildcard semantics: --egress '*.docker.com' covers subdomains at any depth (registry.docker.com and production.cloudfront.docker.com) but not the apex docker.com itself — add it as its own entry. Under the hood k7 translates a leading *. into Cilium's multi-label **. matchPattern; a bare Cilium * never crosses label boundaries, which used to silently break CDN-backed registries (spec 18f issue 3). The install also sets Cilium dnsProxy.minTtl=3600 so clients that cache a resolved IP longer than the CDN's 30–60s DNS TTL (dockerd's blob downloader does) keep their learned FQDN→IP allowance for an hour.

Disk pool sizing (kfd + k7d)

Both node-local storage pools have fixed sizes chosen at install time (spec 18f issue 5) — set them per node in the inventory:

Pool	Backend	Default	Inventory knob	Utilization
kata-vg/thin-pool (LVM, on the spare disk)	kfd	100G PV	kata_thinpool_pv_size	lvs kata-vg (Data%/Meta%)
/var/lib/k7d/disks (sparse XFS loopback)	k7d	32G image	k7d_disks_image_size	df -h /var/lib/k7d/disks

Failure modes when a pool fills — both are invisible to kubelet (no disk-pressure eviction, the pools are not part of the root filesystem):

• kfd thin-pool full: LVM autoextend (thin_pool_autoextend_threshold=80) grows the pool within the PV; once the PV itself is exhausted writes inside sandboxes start failing with I/O errors and new kfd pods fail to create their devmapper snapshots (CreateContainerError). Only the first kata_thinpool_pv_size of the spare disk is used — size it generously.
• k7d pool full: writable volume images and fork reflink clones fail; the k7d daemon rejects new sandboxes/forks loudly (no space left on device). The image is sparse, so ls -l shows the virtual size — use du/df for actual usage.

Utilization for both pools on every node is surfaced through the API (spec 18g) and CLI/SDK (spec 18h): GET /api/v1/nodes/storage, k7 nodes storage (--json for raw), and Client.nodes_storage() return a per-node map of kata_thinpool (lvs size/data%/metadata%) and k7d_disks (df size/used/avail), collected from the k7-agent DaemonSet. A node whose agent is unreachable gets a loud {"error": ...} entry.

Memory limits (--memory)

All three backends honour k7 create --memory <qty> (Kubernetes quantity like 2Gi / 4096Mi):

Backend	Mechanism
kfd	Kata stamps io.katacontainers.config.hypervisor.default_memory (MiB). containerd forwards io.katacontainers.* (pod_annotations on runtimes.kata) and configuration-fc.toml allowlists default_memory (spec 18h — before that the annotation was silently ignored and the VM stayed at the 2048 MiB default).
kql	Same annotation path via runtimes.kata-qemu + configuration-qemu.toml (spec 18g).
k7d	No Kata annotation — the k7d shim sizes the VM straight from the pod's CRI CPU/memory limits.

Known issues

• kata-fc VMM leak under churn (spec 18f issue 1b): kata 3.24.0 with the jailer records the --daemonized jailer's PID, so its SIGTERM fallback at pod deletion signals a dead PID. When the graceful in-guest shutdown fails (dead agent under parallel churn), the firecracker process is orphaned and spins at 100% CPU. k7 install deploys a per-node systemd timer (k7-vmm-reaper.timer, 1-minute cadence) that kills VMM processes whose kata shim is gone, and the integration suite asserts zero orphans cluster wide after teardown (tests/integration/test_zz_leaks.py).
• kql docker-in-VM wedge under heavy fsync + replicas ≥ 2 — FIXED (spec 18f issue 8 / spec 18g): kata's default single-threaded virtiofsd serialized all virtio-fs IO; under a sustained fsync burst against an r≥2 Longhorn volume the kata-agent's health ping starved behind the IO convoy and the shim killed the healthy VM. k7 install now widens the virtiofsd thread pool (--thread-pool-size=16); see PERFORMANCE.md for the root-cause narrative and post-fix numbers.