K8s Agent-Box Runtime

Status: Approved / shipped. Tracking: #840 (umbrella), #841–#845 (sub-issues). The Kubernetes backend is implemented and in CI. See pkg/core/box/k8s/ and docs/KIND-QUICKSTART.md for the local bring-up guide.

What this closes

Today the in-the-box surface (cmd/agent-box — shell + file ops over stdio, SSH-wrapped) only exists on an LXC/incus backend. The transport is SSH: an agent runs ssh <user>@<gateway> -- agent-box and gets a scoped MCP session pinned into a single box. The gateway is sshpiper on the sentinel, routing by SSH username.

There is no equivalent on Kubernetes. Operators who already run a cluster have no way to host an agent box as a pod — they'd have to stand up a separate incus host. This note designs a K8s backend that preserves the exact agent-facing contract (SSH → ForceCommand agent-box → stdio MCP) so an agent cannot tell which runtime it landed on.

This is a backend behind the existing box vocabulary, not a parallel surface. Per the repo's CLI-first convention, the entry point is a --runtime=k8s flag on the existing box-create path; the MCP tool wraps the same Go function.

The agent-facing contract (unchanged)

ssh <tenant>@<gateway-ip> -- agent-box
        │
        ▼  (sshpiper routes by username)
   ForceCommand /usr/local/bin/agent-box   →   stdio MCP loop

Everything below is implementation behind that line. The agent's MCP client, its known_hosts pin, and its scoped key are identical to the LXC path.

Decisions (locked for v1)

Axis	Choice	Rationale
Pod lifecycle	Long-lived per tenant	Mirrors the LXC box model; stable DNS, no cold-start.
SSH ingress	In-cluster sshpiper Deployment	Reuse the sentinel reverse-proxy pattern; per-user fan-out behind one IP.
Isolation	Namespace-per-tenant + default-deny NetworkPolicy	Soft multi-tenancy; the K8s expression of eBPF deny-by-default + egress allowlist.

Ephemeral/pooled lifecycles are explicitly out of scope for v1 — see "Deferred". Hard isolation via RuntimeClass shipped after v1 (#1122, see "Shipped features" below) — gVisor (runsc) works for a box's real, designed traffic (SSH/MCP over pod networking), with one known upstream gap on local debugging access; Kata remains unevaluated.

Topology

                         ns: agent-gateway
                   ┌──────────────────────────────┐
   Agent  ──SSH──▶ │  Service(LB) :22             │
  (MCP cli)        │      │                        │
                   │  sshpiper Deployment          │
                   │  + upstream-controller        │
                   └──────┬───────────────┬────────┘
                          │ route by user │
              ┌───────────▼───┐     ┌─────▼─────────┐
   ns:tenant-a │ Sandbox CR   │     │ Sandbox CR   │ ns:tenant-b
              │ └ pod "box"   │     │ └ pod "box"  │
              │   (sshd +     │     │   (...)      │
              │    agent-box) │     │              │
              │ NetworkPolicy │     │ NetworkPolicy│
              │ default-deny  │     │ default-deny │
              └───────────────┘     └──────────────┘

  (pod + headless Service are created by the kubernetes-sigs/agent-sandbox
   controller from the Sandbox CR; the daemon owns everything else)

Components

1. The box (per-tenant agent-sandbox Sandbox CR)

One kubernetes-sigs/agent-sandbox Sandbox CR (agents.x-k8s.io/v1beta1) per tenant. The agent-sandbox controller — a required install alongside the daemon — creates the pod and a headless Service from it, giving sshpiper a stable DNS name to route to: box.<tenant-ns>.svc.cluster.local (the CR's status.serviceFQDN). The CRD also gives us suspend/resume (spec.operatingMode) and a native absolute expiry (spec.lifecycle.shutdownTime) instead of hand-rolled replica scaling and sweeper-only TTL.

(Originally this was a hand-managed StatefulSet box-0 fronted by a headless Service boxes; replaced when SIG Apps standardized exactly this workload shape as the Sandbox CRD.)

• One container: sshd + the agent-box binary on PATH.
• automountServiceAccountToken: false — the box is a leaf, never a kube-apiserver client. Its authz is the scoped JWT seeded inside, exactly as on LXC.
• securityContext: runAsNonRoot, readOnlyRootFilesystem, capabilities: drop [ALL], seccomp RuntimeDefault.
• The tenant's authorized public key is mounted from a per-tenant Secret (AuthorizedKeysFile), or pulled via AuthorizedKeysCommand from the control plane — the analog of the sentinel POST /authorized-keys/sentinel push. No keys are baked into the image.

sshd_config (image-level):

ForceCommand        /usr/local/bin/agent-box   # every session → MCP stdio loop
AuthorizedKeysFile  /etc/agent-box/authorized_keys
PubkeyAuthentication yes
PasswordAuthentication no
PermitTTY           no
AllowTcpForwarding  no                          # no pivoting out of the box

ForceCommand is the load-bearing line: even a misbehaving client cannot get a shell — it gets agent-box and nothing else.

Interactive-shell mode (opt-in). Setting AGENTBOX_MODE=shell in the container env drops the forced command so a session lands in the agent user's /bin/bash — turning the box into a developer-style SSH machine rather than an MCP-only endpoint. The default (AGENTBOX_MODE=mcp) keeps the forced-command guarantee above. Shell mode deliberately trades that guarantee for a general shell, so anyone who can authenticate gets shell access inside the box: use it only when that's the intent, and pair it with a default-deny NetworkPolicy so the shell cannot reach the cluster network. Everything else — key-only auth, no port forwarding, non-root on :2222 — is unchanged between the two modes.

Shipped image (images/agent-box/): the config above is the contract; the actual image realizes it with dropbear rather than OpenSSH — dropbear runs cleanly rootless and takes a forced command (-c agent-box), so the box runs non-root on :2222 (an unprivileged port → no added capabilities) and the pod satisfies the restricted Pod Security profile. The agent still reaches the gateway on :22; :2222 is only the internal sshpiper→pod hop. Built per release as ghcr.io/footprintai/containarium-agent-box.

2. Gateway (sshpiper Deployment + upstream controller)

sshpiper itself is unchanged from the sentinel deployment — it terminates :22 (via a Service type=LoadBalancer) and routes by SSH username. It stays a dumb L4 reverse proxy; routing state lives one layer up.

The new piece is a thin upstream controller replacing the sentinel's /authorized-keys-poll-and-write-YAML loop:

• Watches tenant objects (CRD or labeled namespaces) + their box pods.
• Programs the sshpiper upstream map via the maintained sshpiper Kubernetes plugin CRD — the Pipe resource (sshpiper.com/v1beta1, plural pipes; earlier drafts of this note called it "PiperUpstream"): spec.from[].username = <tenant> → spec.to.host = box.<tenant-ns>.svc:2222 (the Sandbox's controller-created headless Service; the box's internal SSH port), upstream user agent, with the box's authorized keys inline as authorized_keys_data. The daemon manages Pipes via the dynamic client (no sshpiper Go types imported). CRD-driven removes the file-write race that bit the sentinel (the #301/#404 class of bug).
• Reconciles each tenant's authorized key into the per-tenant Secret.

Credential chain (two keypairs). sshpiper terminates the client connection and opens a new one to the box, so two hops authenticate independently: client→sshpiper against the Pipe's spec.from.authorized_keys_data (the agent's key), and sshpiper→box against spec.to.private_key_secret (sshpiper's upstream key, whose public half the daemon authorizes on the box — the box never authorizes the agent's key in gateway mode). Deployable manifests + runbook live in deploy/k8s/sshpiper/; the daemon is wired via CONTAINARIUM_K8S_GATEWAY_UPSTREAM_{PUBLIC_KEY,KEY_SECRET}.

3. Isolation (NetworkPolicy)

Per tenant namespace:

This section once described an intended design the shipped code did not implement (#1193). Corrected 2026-08-08 after reading pkg/core/box/k8s/objects.go against it, and rewritten as the gaps closed. It now describes what ships, and every claim below is asserted by a test.

The floor: default-deny ingress with a single allow rule — SSH from the sshpiper pod only (matched by agent-gateway namespace + pod label) — and default-deny egress allowlisting cluster DNS and nothing else.

Shipped (one default-deny NetworkPolicy per tenant namespace, networkPolicyObject in objects.go):

• Its podSelector scopes the policy to the tenant's box pods.

• The ingress rule restricts the SSH port to the sshpiper pod — ONE peer carrying both a namespaceSelector (agent-gateway) and a podSelector (app.kubernetes.io/name: sshpiper). One peer, not two: two would be OR, admitting any pod in the gateway namespace or any sshpiper-labelled pod anywhere (#1195).

• The egress rule restricts DNS to the cluster DNS namespace, on both UDP/53 and TCP/53 — TCP matters for large responses.

• The ingress restriction is conditional on GatewayNamespace being set (CONTAINARIUM_K8S_GATEWAY_NAMESPACE, default agent-gateway). Cleared, it means routing is disabled, and the rule falls back to port-only — which admits every source the CNI permits. Emitting a selector that matches nothing instead would make the box unreachable rather than merely unrouted, so the fallback is deliberate; but an operator who clears this variable weakens the ingress boundary and gets no warning. Pinned by TestNetworkPolicyIngressFallsBackWhenGatewayNamespaceUnset, which exists so the fallback stays a choice someone made rather than a regression.

• There is no control-plane API egress rule, and that is now a decision rather than an omission. An earlier draft of this section listed one in the intended allowlist and left the question open pending #1188. #1188 has since shipped: tenant egress is policy-driven, so anything beyond DNS is expressed as a tenant's policy and converged into this same object. A hardcoded control-plane allowance in the floor would grant every box egress to the API server whether or not its policy says so — which is the shape of problem #1193 was filed about, pointed the other way.

  Nothing in the box image needs it: agent-box speaks stdio over SSH, and authorized_keys, host keys and tenant secrets all arrive as mounts. A box can resolve names and reach nothing else, which the k8s e2e observes directly (TestE2E_BoxEgressPosture). A deployment that does need it should say so as a tenant egress policy, where it is visible and revocable.

  The floor being DNS-only is pinned by a test that asserts exactly one egress rule, so adding a second one to the floor fails rather than passing quietly.

The port is sshPort (2222), the internal sshpiper→pod hop; the gateway terminates the operator-facing :22.

This is enforced, not merely declared. kind's default CNI (kindnet) does not implement NetworkPolicy, so for a while the object was created and ignored, and the e2e that "covered" it could not fail (#1234). CI now builds the cluster with Calico and TestE2E_NetworkPolicyIsolation asserts both directions: a pod in another namespace cannot reach the box's SSH port, and — as a positive control that runs first — a pod in agent-gateway carrying the sshpiper label can. Without that control, "connection refused" would prove nothing.

Tenant egress policy on this backend

NetworkPolicyService now drives this backend (#1188): a reconciler reads the same policy store the eBPF enforcer reads and converges each tenant namespace's NetworkPolicy with it. A tenant's egress allowlist appears as egress rules alongside the DNS allowance, and removing a policy reverts to the default-deny floor — never to allow-all, because a delete must not widen access. TestE2E_TenantEgressAllowlistIsEnforced proves the enforcement with packets rather than by inspecting the object: a destination inside the allowlist is reachable, one outside it is not.

What is LXC-only, and why

Some of the eBPF path's capabilities have no Kubernetes NetworkPolicy equivalent. These are not cross-backend, and the compiler REFUSES a policy containing them rather than applying the part it understood — every one of them fails in the same direction, more permissive than the tenant configured, so quietly dropping any would be worse than not supporting the backend at all.

Capability	Why NetworkPolicy cannot express it
LOG_ONLY mode	NetworkPolicy always enforces. Compiling a tenant's dry run would start dropping their traffic. UNSPECIFIED is treated as LOG_ONLY, so this is the default case.
Metadata carve-out (allow_metadata=false)	Denies 169.254.169.254 even when the allowlist covers it — deny-beats-allow. An allow-only policy cannot carve an exception out of an allow rule, and the exception guards cloud credentials.
egress_domains	NetworkPolicy matches IPs, not names. The eBPF path re-resolves on a refresh loop; a resolved snapshot baked into a static object goes stale silently.
Virtual-patch deny rules (#660)	Payload-signature matches in the BPF program. NetworkPolicy is L3/L4 only.
Traffic-flow accounting (#627)	NetworkPolicy has no counters. Needs Hubble/Cilium or equivalent.

A tenant policy using any of these is rejected for the K8s backend with the feature named, so the asymmetry surfaces at configuration time rather than as a silent difference in what is enforced.

Mapping to the existing (LXC) architecture

Containarium (LXC)	This K8s backend
agent-box over stdio, SSH-wrapped	identical — same binary, same ForceCommand
sshpiper on sentinel :22	sshpiper Deployment + LB Service :22
sentinel key-sync → YAML	controller → Pipe CRD + Secret
LXC box per tenant	agent-sandbox Sandbox CR (pod box) per tenant namespace
eBPF deny-by-default + egress allowlist, driven by NetworkPolicyService	static default-deny NetworkPolicy, not policy-driven (#1188), and weaker than documented (#1193)
sshd 2222 (mgmt) vs sshpiper 22	mgmt via kubectl/RBAC; sshpiper owns 22

CLI-first surface

Per the repo convention, the K8s-ness is a backend behind the box vocabulary, not a new verb tree:

containarium box create --runtime=k8s --tenant=<t>

templates the namespace + Sandbox CR + NetworkPolicy + Pipe CRD + per-tenant key Secrets (the controller derives pod + headless Service from the Sandbox). The platform MCP tool wraps the same Go function the CLI handler calls. The backend is selected behind a runtime interface; LXC stays the default.

The box-backend Go interface (sketch)

The seam is the thing that makes --runtime=k8s a backend swap rather than a fork. Two facts about the current code shape where it goes:

1. There is already a pkg/core/incus.Backend interface, but it is a leaky, LXC-shaped seam — Exec, WriteFile, ResolveGPUInputToPCI, GetRawInstance (returns incus config maps), per-key SetConfig over the user.containarium.* namespace. Kubernetes cannot implement that cleanly, and shouldn't have to.
2. SSH addressing + key-sync currently live above the incus backend, in the Manager/jump_server layer (host user + authorized_keys consumed by the sentinel keysync). But those are runtime-specific: LXC uses a host jump-server account; K8s uses a per-tenant Secret + Pipe. So addressing and key-sync must move below the seam.

So the seam sits one altitude above incus.Backend (a coarse, runtime-neutral lifecycle contract) and absorbs SSH identity + addressing. The LXC implementation keeps using incus.Backend internally; the K8s implementation talks to the kube-apiserver. incus.Backend is unchanged.

Core interface

// BoxBackend is the runtime-neutral seam. LXC/incus and Kubernetes both
// implement it. Coarse-grained on purpose — no Exec/WriteFile/config-key
// leakage. ctx is threaded (the K8s client needs it; LXC ignores it).
type BoxBackend interface {
    Kind() BackendKind // "lxc" | "k8s"

    // Lifecycle. Create is declarative: given a runtime-neutral spec, make
    // the box exist and return a handle. Idempotent on re-create (the #669
    // lesson from the agent-skills bring-up).
    Create(ctx context.Context, spec BoxSpec) (*BoxHandle, error)
    Start(ctx context.Context, ref BoxRef) error
    Stop(ctx context.Context, ref BoxRef, force bool) error
    Delete(ctx context.Context, ref BoxRef, force bool) error

    // Introspection.
    Get(ctx context.Context, ref BoxRef) (*BoxStatus, error)
    List(ctx context.Context) ([]BoxStatus, error)

    // Addressing — how an agent reaches this box over SSH. THE method that
    // makes the K8s value real. LXC returns sentinel-host|IP; K8s returns
    // the gateway LB host + the username sshpiper routes by.
    Resolve(ctx context.Context, ref BoxRef) (*BoxEndpoint, error)

    // SSH identity. Below the seam because the mechanism differs per runtime:
    // LXC writes the host jump-server authorized_keys; K8s reconciles the
    // per-tenant Secret + Pipe.
    SetAuthorizedKeys(ctx context.Context, ref BoxRef, keys []string) error

    // Mutation. Meta is the runtime-neutral replacement for raw incus config
    // keys: TTL, delete-policy, labels, monitoring-enabled. LXC maps it to
    // user.containarium.*; K8s maps it to pod annotations/labels.
    Resize(ctx context.Context, ref BoxRef, r ResourceLimits) error
    SetMeta(ctx context.Context, ref BoxRef, meta map[string]string) error
    GetMeta(ctx context.Context, ref BoxRef) (map[string]string, error)
}

Runtime-neutral types

type BoxRef struct {
    Tenant string // routing key / SSH username
    Name   string // box name (LXC: "<tenant>-container"; K8s: Sandbox "box")
}

type BoxSpec struct {
    Ref        BoxRef
    Image      string
    OSType     pb.OSType
    Resources  ResourceLimits   // cpu / memory / disk
    GPUs       []string         // empty on K8s v1 (deferred)
    SSHKeys    []string
    Labels     map[string]string
    Monitoring bool
    // Provisioning intent — NOT an Exec script. The backend decides how to
    // realize it: LXC runs incus exec; K8s bakes it into the image / an
    // init container. Keeps stack-install runtime-specific, below the seam.
    Stack       string
    StackParams map[string]string
}

type BoxHandle struct {
    Ref      BoxRef
    Endpoint BoxEndpoint
    State    pb.ContainerState
}

type BoxEndpoint struct {
    // The server turns this into Container.ssh_host + the ssh command.
    SSHHost    string // gateway/sentinel public host ("" = direct IP mode)
    SSHPort    int    // 22 via sshpiper
    SSHUser    string // routing key for sshpiper (the tenant)
    DirectIP   string // fallback when no gateway is in front
    AccessType pb.AccessType
}

type BoxStatus struct {
    Ref       BoxRef
    State     pb.ContainerState
    Endpoint  BoxEndpoint
    Resources ResourceLimits
    Meta      map[string]string // TTL, delete-policy, labels, monitoring…
    BackendID string
}

Capability interfaces (not every backend supports everything)

Optional surfaces stay off the core interface and are discovered by type assertion, so a backend only implements what it can honor:

// In-box exec/file seed. LXC implements it (incus exec / file push). The K8s
// agent-box uses ForceCommand, so v1 may NOT implement this — provisioning is
// image-baked. Callers must handle the unsupported case.
type ExecCapable interface {
    Exec(ctx context.Context, ref BoxRef, cmd []string) (stdout, stderr string, err error)
    WriteFile(ctx context.Context, ref BoxRef, path string, content []byte, mode string) error
}

type MetricsCapable interface {
    Metrics(ctx context.Context, ref BoxRef) (*BoxMetrics, error)
}

type GPUCapable interface { // LXC v1; K8s deferred
    ResolveGPU(ctx context.Context, input string) (deviceID string, err error)
}

What stays ABOVE the seam (runtime-neutral, unchanged)

These keep living in ContainerServer / a slimmed Manager and call the backend — they are not duplicated per runtime:

• Auth (JWT scope checks), peer routing (PeerPool, backend_id fan-out), async create tracking (PendingCreation), event emission.
• Cascade cleanup orchestration (routes, TLS subjects) — though the teardown of SSH identity now flows through Delete + SetAuthorizedKeys.
• The proto ↔ domain mapping (toProtoContainer reads BoxStatus, not incus config maps directly).

Wiring

ContainerServer holds a BoxBackend instead of a concrete Manager. The runtime is chosen at daemon start (flag/env) and per-request via --runtime → BoxSpec; incus.Backend continues to back the LXC implementation untouched. The first refactor PR introduces BoxBackend + the LXC implementation as a pure wrapper over today's Manager (no behavior change, golden test parity), and only then lands the K8s implementation against the same contract.

Packaging & repo strategy — one binary, runtime selection

The K8s backend is always compiled into the daemon binary (no build tag). client-go is already in go.mod; the dependency cost is accepted in exchange for a simpler build surface. The active backend is selected at daemon start:

CONTAINARIUM_RUNTIME=k8s containarium daemon start   # Kubernetes backend
CONTAINARIUM_RUNTIME=lxc containarium daemon start   # LXC/incus backend (default)

or via the flag: containarium daemon start --runtime=k8s.

See internal/server/boxbackend_factory.go for the factory. The interface lives in pkg/core/box (public, not internal/) so a future out-of-process bridge can import it without promoting internals.

Future: out-of-process bridge

The multi-backend-peer / backend_id routing already exists. The natural end-state is a K8s bridge that registers like a peer — client-go then leaves the core dependency graph entirely. That is a Phase 2 move, after the BoxBackend interface stabilizes.

Per the OSS/Cloud convention, the bridge is a generic mechanism → it ships in OSS, wherever it lives. BYO-cluster support/packaging may be a commercial concern; the backend itself is not task-specific.

Why a K8s operator would want this

The pitch is not "another way to run pods" — Kubernetes already runs pods. It is: give an AI agent a safe, SSH-native foothold in your cluster without handing it kubectl or a kube-apiserver token.

• No kube-apiserver credential in the agent's hands. The agent reaches a box over SSH and gets a ForceCommand-pinned agent-box stdio MCP — never a cluster client. The box runs with automountServiceAccountToken: false. The blast radius of a compromised agent is one hardened pod, not the cluster API. kubectl exec-based agent access, by contrast, requires RBAC that almost always over-grants.
• Passes restricted PodSecurity as-is. Non-root, drop-ALL, seccomp RuntimeDefault, read-only rootfs. It is a better-behaved tenant than most workloads — installs cleanly on locked-down clusters.
• Auditable, namespaced RBAC footprint. A reviewable Helm chart + a namespaced ServiceAccount. No cluster-admin at steady state (only a one-time CRD install). Security teams can reason about exactly what it can touch.
• In-kernel egress isolation, the K8s-native way. Default-deny NetworkPolicy + egress allowlist — the same deny-by-default posture shipped via eBPF on the LXC backend, expressed in primitives the cluster already enforces.
• Bring your own cluster, your own nodes, your own GPUs. No new control plane to run, no second scheduler. The agent foothold lives next to the data and the GPUs it needs, inside the network boundary the operator already trusts.
• One agent contract across runtimes. The same agent-box surface, the same scoped-JWT model, the same CLI (containarium box create) whether the box lands on LXC or K8s. Teams standardize the agent interface once and pick the substrate per environment.
• The access path survives hard isolation. The standard way most agent-sandbox tooling reaches a pod — kubectl exec / kubectl port-forward — does not work against a gVisor (runsc)-scheduled pod (kubernetes-sigs/agent-sandbox#158, "planned to be supported later," no committed timeline). An operator who wants gVisor's kernel boundary and a working access path is stuck choosing one. Containarium's box access was never built on that mechanism — it's SSH through the sshpiper gateway over real pod networking (see "Hard isolation via RuntimeClass" below) — so turning on runsc costs nothing on the access side; #1489 tracks the one gap (direct port-forward to the box pod, which nothing here depends on).

In one line: the safest blast-radius for an autonomous agent in your cluster — SSH-native, RBAC-minimal, default-deny — with zero new control plane.

Adoption plan (platform engineers / installs)

Buy-in target: platform / DevOps engineers, success metric: adoption (installs/stars). That collapses the whole campaign onto one thing — time-to-wow on kind, then maximal discoverability. Security depth and CNCF credibility are supporting assets, not the lead.

The governing number: time-to-wow

Platform engineers evaluate with helm install and a timer. The funnel is:

find it → kind up → helm install → ssh agent@localhost -- agent-box → "oh, nice"

Under ~5 minutes and copy-pasteable → installs. Needs a real cluster, a cloud account, or hand-edited YAML → no installs. Launch definition of done:

• kind create cluster → helm repo add → helm install → working agent box, zero manual YAML edits.
• Ships a NetworkPolicy-enforcing CNI in the kind config — a kind default does not enforce NetworkPolicy, so the isolation demo would silently no-op.
• The "wow" beat is built into the quickstart, not buried: right after the agent does a task, three one-liners that show no SA token / egress dropped / shell refused. Safety is demonstrated, not asserted, in the same five minutes.

Discoverability (where platform engineers find tools), in priority order

1. Artifact Hub — the Helm chart. The search surface for "is there a chart for X." Non-negotiable for an installs metric.
2. README leads with the problem + a copy-paste install above the fold — the install command visible without scrolling; the one-line problem statement directly above it.
3. awesome-kubernetes / awesome-mcp / awesome-kubernetes-security list PRs — the upstream-list traffic play, retargeted.
4. asciinema/GIF of the 5-min demo embedded in the README — proof-of-wow before they install.
5. CNCF Landscape entry — low urgency for raw installs, cheap, lends legitimacy that converts skeptics.

Sequencing

1. P0 — artifact: Helm chart + kind quickstart hitting time-to-wow + above-the-fold README. Nothing ships until this is real.
2. P0 — discoverability: Artifact Hub + asciinema + awesome-list PRs, same week as the chart.
3. P1 — trust backstop: threat-model doc + Gateway API / NetworkPolicy / PSA conformance, linked from the README.
4. P2 — compounding credibility: problem-framed blog post, CNCF Landscape, public dogfood.

The risk for an installs goal specifically

SSH-transport weirdness causes bounce before the threat model is read — "SSH? in K8s?" → tab closed. For an adoption play the mitigation is not a doc, it's the demo doing the convincing: the no SA token / egress dropped beat must land in the same screen as the install, because for this audience seeing-is-believing beats prose. The threat model is a backstop you link to, not the front door.

Integrating with an existing cluster (BYO)

The primary target is not a dedicated cluster — it is a cluster the customer already owns, where Containarium's control plane manages boxes through scoped access it is granted. This is the K8s analog of the multi-backend-peer / remote-connector pattern: the control plane drives a cluster it does not own, via a namespaced operator running under its own ServiceAccount — never via the operator's personal credentials.

Design the operator for BYO and the own-cluster case falls out for free: BYO is the strict superset of constraints.

What the box demands of the host cluster (it's modest)

The box is a hardened leaf, not a cluster client: non-root, automountServiceAccountToken: false, readOnlyRootFilesystem, drop [ALL], seccomp RuntimeDefault, no host mounts, no privileged caps. It satisfies the PodSecurity restricted profile as-is, so it passes admission on locked-down clusters cleanly. That is a feature: the RBAC ask is small and auditable.

Controller RBAC footprint (namespaced, not cluster-wide)

Verb scope	Resources	Boundary
create/get/delete	Sandbox (agents.x-k8s.io), Secret, NetworkPolicy, PVC	label-selected tenant namespaces only
(controller-owned)	Pod, Service	created by the agent-sandbox controller from the Sandbox, not by the daemon
CRUD	Pipe	gateway namespace only
create	Namespace	only if the controller owns tenant-namespace lifecycle

Shipped as a Helm chart / operator bundle the customer reviews and installs. Containarium then drives the cluster through the controller's ServiceAccount.

Hard gates — absent these, the design changes (not just config)

Requirement	Why	Fallback if missing
L4/TCP ingress (LoadBalancer / NodePort / Gateway API TCPRoute)	SSH is TCP; K8s Ingress is HTTP-only	NodePort + external LB; kubectl port-forward for dev
NetworkPolicy-enforcing CNI (Calico, Cilium, …)	default-deny isolation is a no-op under a CNI that ignores it	degrade to namespace-only isolation — must flag loudly
CRD install rights (cluster-admin, once) for Pipe	the sshpiper Kubernetes plugin is CRD-driven	yaml plugin — re-inherits the sentinel file-write race
Namespace-create rights for the controller	namespace-per-tenant	pin to one shared namespace + label/pod-selector separation (weaker)

Degraded modes (named, not silent)

• No LoadBalancer (bare-metal, no MetalLB) → NodePort + documented external LB.
• PodSecurity restricted enforced → box already complies; no action.
• Single shared namespace mandated → drop namespace-per-tenant; rely on NetworkPolicy pod-selectors + per-box ServiceAccount. Weaker blast radius — call it out at create time.

The make-or-break decision

Whether the customer grants a one-time cluster-admin CRD install. With it, the CRD kubernetes plugin gives clean, race-free upstream programming. Without it, the yaml plugin works but re-inherits the file-write race (#301/#404 class). Document CRD-install as the happy path, yaml as the explicit fallback — and surface which mode is active in box status.

Open questions

1. sshpiper plugin — CRD kubernetes plugin (eliminates the file-write race) vs. the yaml plugin run today. Leaning CRD; the daemon already manages Pipe objects via the dynamic client.
2. Host-key trust — pre-distribute the gateway host key (ConfigMap → agent known_hosts) so first-connect is not a TOFU prompt.
3. ExecCapable — K8s v1 does NOT implement it (provisioning is image-baked; ForceCommand pins the session). Callers discover support via type assertion.
4. GPU node affinity — the gpu-spec label → node affinity mapping (so the scheduler picks the right GPU node pool) is not yet wired.

Shipped features

Runtime selection (#842)

The daemon binary ships one factory that supports both backends:

# Default: LXC/incus (unchanged behaviour)
containarium daemon start

# Kubernetes backend
CONTAINARIUM_RUNTIME=k8s containarium daemon start --runtime=k8s \
  --skip-infra-init \
  --standalone

--runtime takes precedence over CONTAINARIUM_RUNTIME. On a K8s host with no incus installed, the daemon starts cleanly; box lifecycle goes through the kube-apiserver; incus-only RPCs (Exec, GPU resolve, core-container detection) return clear errors.

Key env vars for the K8s backend. These are read once at daemon start through the typed internal/config loader: config.LoadK8s() returns a config.K8s (whose Env* constants are the single source of truth for the names below, and which applies the defaults shown), and config.K8s.Validate() fails fast on a bad gateway port. The server factory (newK8sBackend) maps the result onto the env-agnostic pkg/core/box/k8s.Config — so pkg/core reads no environment.

Env	Default	Purpose
CONTAINARIUM_K8S_KUBECONFIG	ambient rules	Path to kubeconfig; empty = in-cluster
CONTAINARIUM_K8S_BOX_IMAGE	(required)	Agent-box image (ghcr.io/footprintai/containarium-agent-box)
CONTAINARIUM_K8S_GATEWAY_HOST	(required)	Public SSH gateway host (sshpiper LB)
CONTAINARIUM_K8S_GATEWAY_SSH_PORT	22	Gateway SSH port surfaced on the box endpoint
CONTAINARIUM_K8S_GATEWAY_NAMESPACE	agent-gateway	Namespace sshpiper runs in
CONTAINARIUM_K8S_TENANT_NS_PREFIX	tenant-	Prefix for per-tenant namespaces
CONTAINARIUM_K8S_STORAGE_CLASS	(empty = no PVC)	StorageClass for persistent data
CONTAINARIUM_K8S_GATEWAY_UPSTREAM_PUBLIC_KEY	(empty)	Public key sshpiper→box authenticates with. Required when GATEWAY_NAMESPACE is non-empty — see below
CONTAINARIUM_K8S_GATEWAY_UPSTREAM_KEY_SECRET	(empty)	Secret name holding the matching private key. Required when GATEWAY_NAMESPACE is non-empty (#1496): with gateway routing enabled and no upstream credential, sshpiper falls back to password auth, which every box refuses — K8s.Validate() refuses daemon startup rather than produce that silently-broken Pipe. Clear GATEWAY_NAMESPACE to disable gateway routing and drop this requirement
CONTAINARIUM_K8S_INSECURE_IGNORE_HOST_KEY	0	1 skips box host-key pinning (escape hatch, not recommended)
CONTAINARIUM_K8S_DEFAULT_MEMORY_REQUEST	256Mi	Default per-box memory request when the box sets none; invalid → built-in default
CONTAINARIUM_K8S_DEFAULT_MEMORY_LIMIT	1Gi	Default per-box memory limit (hard cap, noisy-neighbor guard); invalid → built-in default
CONTAINARIUM_K8S_DISABLE_MEMORY_FLOOR	0	1 disables the floor — boxes with no explicit memory run unconstrained

CSI persistent storage (#841 / #844)

Each box optionally gets a PersistentVolumeClaim for a persistent working directory, mounted at /home/agent/workspace — a subdirectory of the image's home directory, not the home directory itself. Controlled by CONTAINARIUM_K8S_STORAGE_CLASS:

• Empty (default): no PVC; namespace is deleted on Delete (original behavior, backward-compatible).
• Non-empty: PVC named data is created before the Sandbox and mounted as a plain pod volume, not a volumeClaimTemplate — template-derived PVCs are owner-referenced to the Sandbox and garbage-collected with it, which would break this contract. Delete removes compute objects (the Sandbox — cascading to pod + Service — plus NetworkPolicy, Secrets) but retains the namespace + PVC so data survives a node reap. Purge removes both PVC and namespace when the tenant is gone.

This design lets autoscaled GCE VMs be reaped without data loss — the PV is CSI-managed (GKE, EKS, or kind local-path) and outlives any compute node.

Disk size is read from BoxSpec.Resources.Disk (e.g., "20Gi"); defaults to 10Gi when unset.

Mount path is /home/agent/workspace, not /home/agent (#974). An earlier revision mounted the PVC directly over /home/agent. That replaced the box image's real home directory with the provisioner's fresh volume root — 0777 root:root on kind/local-path, 0755 root:root on a typical CSI ext4 root — either of which broke SSH: dropbear's strict modes check rejects a group/world-writable home directory outright, and a 0755 root:root root additionally leaves uid-1000 agent unable to write (or in some cases even read) its own home. Every login to a storage-backed box failed with Permission denied (publickey) even though the authorized_keys Secret was mounted and matched the client key. Neither the unit tests nor the kind e2e caught it because they asserted the PVC existed and was mounted, never that SSH auth actually succeeded afterward.

The fix mounts the PVC one level down, at /home/agent/workspace, leaving /home/agent itself exactly as the image built it — owned by agent, dropbear-compatible permissions, with the authorized_keys Secret layered in via its own separate volume mount (unaffected by this change either way).

Data-contract change: before #974, the entire home directory persisted across box recreation on storage-backed boxes. After #974, only /home/agent/workspace persists — files written elsewhere under /home/agent (e.g. shell rc files, caches outside the workspace dir) reset to the image defaults on recreate, the same as on non-storage-backed boxes. Agents and tooling that assumed the whole home directory was durable on K8s storage-backed boxes should keep persistent state under /home/agent/workspace.

GPU resource requests (#845)

BoxSpec.GPUs []string maps to a nvidia.com/gpu extended-resource limit on the box container:

len(spec.GPUs) > 0  →  container.Resources.Limits["nvidia.com/gpu"] = N

The K8s cluster autoscaler uses this to scale up a GPU node pool when no schedulable node is available — no Containarium-side autoscaler needed for the K8s backend. The pod template carries a containarium.dev/gpu-count: "N" annotation for observability.

The GPU type (L4, A100, etc.) is expressed via node affinity, driven by a gpu-spec label on the box when set — otherwise K8s schedules to any GPU node. This is deliberately different from the LXC/GCE path, where the daemon selects the exact machine type; on K8s, the scheduler owns that decision.

Hard isolation via RuntimeClass (#1122)

Config.RuntimeClass (CONTAINARIUM_K8S_RUNTIME_CLASS / chart runtimeClass) sets RuntimeClassName on every box pod. Empty (default) leaves it unset — pods run on runc, sharing the host kernel, byte-identical to pre-#1122 behavior. Set to runsc to schedule boxes behind a gVisor sandbox instead, on a node pool where gVisor is installed and a RuntimeClass named runsc exists. Daemon-wide, not per-box: the runtime is a property of the node pool the daemon schedules onto.

Verified working (live kind cluster with runsc installed as a containerd runtime handler, manual QA 2026-08-22, following up on this PR): pod genuinely runs on the gVisor kernel; SSH/dropbear handshake and the ForceCommand pin; a full MCP initialize round-trip; shell_exec (fork/pipe/exec); write_file; PVC-backed storage permissions (identical to runc); and default-deny NetworkPolicy enforcement (both directions) — all pass over the box's real, designed traffic path (SSH/MCP over pod networking, the same path the sshpiper gateway uses in production).

Recorded 2026-08-22 on a local kind+gVisor test cluster. Shows kubectl port-forward failing against the runsc box, then a real SSH session completing an MCP initialize and a shell_exec(cat /proc/version) call against the box's actual K8s Service DNS name — real pod-to-pod traffic, not a port-forward or a kubectl exec. The recording does not include the sshpiper gateway hop itself: that hop hit a separate, still-unresolved authentication failure during this same verification pass (#1493), so this clip demonstrates the box's reachability and function under runsc, not a fully gateway-through-to-box round trip.

Known gap: kubectl port-forward dialed directly against a runsc-scheduled box pod does not work — connection refused, even though the same port is fully reachable over real pod-to-pod networking (demonstrated above). This is an upstream gVisor/kubelet characteristic, not a Containarium defect: the exact same failure is documented independently at kubernetes-sigs/agent-sandbox#158 ("planned to be supported later" — no committed timeline as of this writing). Tracked here as #1489.

Correction (2026-08-22): the line originally here also claimed kubectl exec does not work against a runsc box. Re-verified directly against a live runsc box and that claim is wrong — kubectl exec (spawning a fresh process: id, uname -a, ls, an interactive -it shell) works normally under gVisor every time it was tried. Only kubectl port-forward — dialing an existing listening socket from outside the sandbox — fails. The distinction matters: exec doesn't touch the sandboxed network stack at all, so it was never actually exposed to the same limitation.

Separately, this same verification pass found the Helm-chart-deployed gateway path has two more issues, neither specific to gVisor: the chart's default-deny NetworkPolicy can't actually match the chart's own sshpiper pod (#1492, label mismatch — blocks real traffic regardless of runtime class), and even after fixing that, sshpiper's own pubkey check rejected a client key that byte-for-byte matched what was registered (#1493, root cause not yet isolated). So "the same path the sshpiper gateway uses in production" below is not yet independently confirmed working end-to-end on the Helm-chart path — the recording above validates the box's own reachability and correctness under runsc, using a workaround for the gateway hop specifically.

Update (2026-08-22, later the same day): #1492 and #1493's root cause (#1496 — the Helm chart's default has no upstream keypair configured, so sshpiper falls back to password auth) are both fixed and merged (PR #1495). The gateway path was then re-verified for real: a fresh cluster, main at the tip (including #1495), deployed via helm install exactly as KIND-QUICKSTART.md's Helm quickstart now documents — no workarounds, no stand-in pods.

Recorded 2026-08-22 on a fresh kind+gVisor cluster, main @ commit including #1495. sshpiper's own log for this session: ssh connection pipe created ... (username [mybox]) -> ... (username [agent]) — a real pipe, authenticated with the configured upstream keypair (auth [privatekey]), not the password fallback #1496 was about. The client then completes a full MCP initialize + shell_exec(cat /proc/version) round trip through that pipe, returning Linux version 4.19.0-gvisor — proving the box, the gateway, and gVisor all work together, for real, end to end.

"The same path the sshpiper gateway uses in production" is now independently confirmed working end-to-end on the Helm-chart deployment. The correction above is left in place rather than deleted — it was accurate when written, and the gap it named is exactly what #1495 closed.

One more gap found in the process, unrelated to gVisor or #1492/#1496: the Helm quickstart never told operators to create sshpiper's server key Secret (sshpiper-server-key) — only the upstream one. Missing it doesn't fail loudly like #1496 did; the sshpiper pod just sits in ContainerCreating forever on a FailedMount event, easy to miss. Fixed in KIND-QUICKSTART.md alongside this recording.

Practical consequence: never reach a gVisor box by port-forwarding straight to its pod — that upstream gVisor/kubelet gap (#1489) is unaffected by anything above and still applies. Go through the sshpiper gateway instead (a NodePort/LoadBalancer, or a port-forward to svc/sshpiper itself, which is not gVisor-scheduled) — confirmed working end-to-end as of the update above, on the documented Helm-chart deployment path. See KIND-QUICKSTART.md and deploy/k8s/sshpiper/README.md for the gateway-based access path.

Tenant secret delivery (#1190)

A tenant's secrets are materialized into a per-tenant Secret and mounted into the box at /run/secrets, one file per secret — the same path the LXC file-delivery mode uses. The box's session builds its environment from those files, so an env-delivery secret behaves the same on both backends.

Why a mount and not env vars. A container's environment is fixed when it starts. Updating a Secret projected with envFrom does not change a running pod — it has to be recreated. A Secret volume is refreshed in place by the kubelet, so a new session sees the new value with no restart. That is also what the LXC path actually promises: its refresh message says new execs see updated values, not that running processes are re-parented. The mount is what makes the two backends agree.

Because the mount is refreshed by the kubelet and survives a restart, the LXC-side secrets reconciler has no K8s counterpart and is not wired there. It exists because a container's tmpfs does not survive a restart; a mounted Secret does.

All three delivery modes land in the same object. env and file rows each become one file; compose rows are rendered into a single secrets.env by the same function the LXC path uses, so a compose app's env_file: reference differs only in directory.

Boxes created before this shipped. The secrets volume is part of the box's pod template, so a box created before it existed has no mount and no amount of delivery will reach it — the Secret is written and nothing consumes it. Those boxes need recreating. This fails quietly rather than loudly: secret refresh reports success, because the delivery genuinely succeeded; it is the mount that is missing. Recreate a box and check for /run/secrets inside it if secrets appear to be ignored on a long-lived deployment.

Operator responsibility: encryption at rest. A Kubernetes Secret is base64-encoded, not encrypted, unless the cluster has encryption at rest configured for the secrets resource. Containarium's own at-rest guarantee — envelope encryption with a KMS-held KEK — covers the value in Postgres, and it still does on this backend. It does not extend to etcd.

So a tenant moving from the LXC backend to K8s gets a weaker at-rest guarantee for the delivered copy unless the operator configures it, and the platform cannot detect or enforce that from inside. Stated here rather than left implied: a reader who assumes the KMS guarantee covers the whole path would be wrong, and would be wrong silently.

Anyone can read a mounted secret who can kubectl get secret in the tenant namespace, kubectl exec into the box, or read etcd directly. The first two are the same trust boundary as the LXC backend (an operator with incus access can read a container's config and its tmpfs); the third is new, and is what cluster encryption at rest addresses.

Fronting a K8s node with the fleet sentinel

Everything above describes a K8s node reached at its own in-cluster gateway (ssh <box>@<node-gateway>). In a multi-node fleet, the Containarium sentinel is the single public SSH entrypoint and chains to each node's gateway — so ssh <box>@<sentinel> reaches a box on any node, K8s or LXC, with the client holding only its agent key:

agent ──agent key──▶ sentinel sshpiper :22
                         │  (keysync: username → node:<ssh_port>)
                         ▼
        node in-cluster sshpiper (NodePort/LB)
                         │  (Pipe: username → box pod, node upstream key)
                         ▼
                   box pod :2222 (dropbear ForceCommand → agent-box MCP)

Three hops; each key authenticates exactly one:

• agent → sentinel: the agent's key, in the sentinel's per-user authorized_keys (synced from the node's /authorized-keys).
• sentinel → node gateway: the sentinel's upstream key, authorized at the node gateway (appended to every box's Pipe from by the daemon when the sentinel POSTs /authorized-keys/sentinel). Requires gateway-upstream mode.
• node gateway → box: the node's own upstream key (the box authorizes it).

How the node participates:

• Start the daemon with --ssh-host <sentinel> so boxes surface the sentinel as their ssh_host (stamped runtime-neutrally, same as LXC).
• The node advertises its gateway ingress port to the sentinel automatically (/authorized-keys ssh_port, resolved from the gateway Service NodePort or CONTAINARIUM_K8S_GATEWAY_ADVERTISE_PORT).
• Attach to the sentinel like any backend: direct (routable node IP) or via the yamux tunnel. A tunnel-attached node forwards its gateway port to the Service's reachable address — containarium tunnel --forward <port>=<addr>; see TUNNEL-REVERSE-PROXY.md.

End-to-end verification: scripts/k8s-sentinel-e2e.sh (kind node gateway + a stand-in sentinel sshpiperd + a two-hop MCP handshake).

Sandbox-CRD semantics worth knowing

• Stop/Start = suspend/resume. Stop patches spec.operatingMode: Suspended — the controller deletes only the pod; PVC, Service, Secrets, and the Sandbox identity persist. Start patches back to Running and the pod is recreated with the same PVC.
• Resize restarts the pod. The agent-sandbox controller does not restart a live pod on podTemplate drift, so after patching resources the daemon bounces a running box (suspend → resume) to apply them now — comparable downtime to the StatefulSet rolling restart the old path triggered.
• TTL is two mechanisms in one patch. SetContainerTTL stamps the Sandbox's spec.shutdownTime with shutdownPolicy: Retain (the controller stops the pod at the deadline even if the daemon is down) plus a mirrored ttl_expires_at meta annotation that the daemon's TTL sweeper reads; the sweeper routes the actual delete through the full DeleteContainer cascade. Retain — not Delete — because a controller-side Sandbox delete would orphan the daemon-owned Pipe, Secrets, PVC, namespace, and routes.

Migrating a pre-Sandbox deployment

Deployments created by the StatefulSet-era backend must recreate their boxes (the daemon no longer manages the old object shape):

1. On the old build: containarium container delete <tenant> per box — or, post-upgrade, kubectl delete statefulset box; kubectl delete svc boxes in each tenant-* namespace.
2. Upgrade the daemon; ensure the agent-sandbox controller is installed.
3. containarium container create again. With a StorageClass configured, home data survives: the new backend reuses the same daemon-owned PVC data as a plain pod volume.

The chart's ClusterRole keeps the legacy apps/statefulsets rule for one release so the post-upgrade cleanup in step 1 works, then it gets dropped.

Deferred (not in v1)

• Kata RuntimeClass — gVisor shipped instead (#1122, see "Shipped features"); Kata (a VM-per-pod boundary rather than gVisor's userspace kernel) remains unevaluated.
• Ephemeral / pooled lifecycles — spin-up-on-connect or warm-pool leasing. agent-sandbox ships SandboxTemplate/SandboxWarmPool/SandboxClaim for this, but claim adoption is same-namespace-only (ErrCrossNamespaceAdoption), so warm capacity cannot be shared across per-tenant namespaces — a shared pool would require collapsing the tenancy model, and per-tenant pools invert the economics (N tenants × idle warm replicas). Until that tension is resolved (upstream cross-namespace pools, or a tenancy redesign), most of the fast-create win comes from pre-pulling the agent-box image (DaemonSet) + suspend/resume, since resume is just pod creation on a warm node.
• Cross-cluster / multi-pool fan-out (the K8s analog of multi-backend peers).
• GPU node affinity by spec — the gpu-spec label → node affinity mapping is not yet wired; scheduling to any GPU node is the v1 behavior.