⬡ RunMesh

April 14, 2026 · View on GitHub

⬡ RunMesh

Production-grade multi-tenant sandbox execution plane for autonomous AI agents.

Schedule · isolate · execute · observe — at any scale.

Overview

Runtime Fabric is a multi-tenant sandbox execution plane purpose-built for AI agent workloads. It provides:

Sub-500 ms cold starts via warm-pool pre-provisioning and eager image caching
Hard multi-tenant isolation using gVisor kernel interception + OCI sandbox abstractions
Production-grade scheduling with best-fit bin-packing across a dynamic node fleet
Native agent primitives — streaming exec, ephemeral storage, and multi-step task chaining
Full observability — Prometheus metrics, OpenTelemetry distributed tracing, and a glassmorphic real-time dashboard

It is the missing execution layer beneath your LLM agent framework. Give Fabric an OCI image and a prompt; it handles everything from placement to cleanup.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Client / API Layer                      │
│              gRPC (sandbox.proto)  ·  REST (webhook)         │
└──────────────────────────┬──────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────────┐
│                     Control Plane (Go)                       │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │  API Server  │  │  Scheduler   │  │  Chain Executor  │   │
│  │  (webhook +  │  │  (bin-pack)  │  │  (task sequencer)│   │
│  │   REST gate) │  └──────┬───────┘  └──────────────────┘   │
│  └──────┬───────┘         │                                  │
│         │        ┌────────▼──────────────────────┐          │
│         └───────►│   State Store  (etcd v3)       │          │
│                  │   leader election · IPAM ·      │          │
│                  │   warm-pool registry             │          │
│                  └────────┬──────────────────────-─┘          │
└───────────────────────────┼─────────────────────────────────┘
                            │ gRPC dispatch
          ┌─────────────────┼─────────────────────────┐
          │                 │                          │
┌─────────▼──────┐ ┌────────▼───────┐ ┌───────────────▼──┐
│  Node Agent    │ │  Node Agent    │ │   Node Agent     │
│  (Rust)        │ │  (Rust)        │ │   (Rust)         │
│                │ │                │ │                  │
│ containerd     │ │ containerd     │ │  containerd      │
│ gVisor (runsc) │ │ gVisor (runsc) │ │  gVisor (runsc)  │
│ eBPF monitor   │ │ eBPF monitor   │ │  eBPF monitor    │
└────────────────┘ └────────────────┘ └──────────────────┘

Component Map

Package	Language	Responsibility
`control-plane/`	Go	API server, scheduler orchestration, resource state, IPAM
`scheduler/`	Go	Node registry (etcd leases), bin-pack engine, gRPC dispatcher
`node-agent/`	Rust	Container runtime, streaming exec, eBPF security, ephemeral storage
`api/proto/`	Protobuf	Shared gRPC service definitions
`pkg/`	Go	Shared utilities and types
`cli/`	Go	Developer CLI (`fabric run`, `fabric logs`)
`deploy/`	YAML	Kubernetes manifests and Helm chart
`dashboard/`	HTML/JS	Real-time cluster observability UI

Key Features

🗓 Best-Fit Bin-Packing Scheduler

The scheduler (scheduler/internal/scheduler/engine.go) implements a best-fit-decreasing algorithm:

Filter nodes by available CPU + memory (including configurable overcommit ratio)
Prioritise nodes that already have the requested image in their warm-sandbox pool
Score remaining candidates by remaining capacity and place on the densest viable node

This maximises utilisation while ensuring no node becomes a bottleneck.

🔒 Hard Tenant Isolation

Every sandbox runs under gVisor (gvisor / runsc runtime class) providing:

Kernel interception — the container never touches the host Linux kernel directly
seccomp + eBPF syscall filtering via the node-agent security module
Network namespace isolation (per-sandbox CNI attachment)
Tenant ID enforcement on all etcd reads/writes via prefix-scoped transactions

Fabric can also target alternate runtime classes when a node advertises them, including microsandbox for local VM-isolated execution through the msb CLI.

♻️ Warm Pool & Pre-provisioning

The control plane maintains a pool of pre-started, idle sandboxes keyed by (image, runtime_class). Incoming requests first check the pool via an etcd transaction (exactly-once claim), slashing cold-start latency from ~1-2 seconds to < 300 ms for popular images.

⛓ Task Chaining (Agent Workflows)

control-plane/internal/chain/executor.go sequences multi-step agentic workflows:

Step 1 (sandbox A) → result → Step 2 (sandbox B) → result → Step 3 ...

Each step creates a fresh, isolated sandbox; the chain executor passes outputs between steps automatically and cleans up on success or failure.

📡 Streaming Exec (`ExecCommand`)

After a sandbox is running, clients can ExecCommand — a gRPC server-streaming RPC that:

Accepts arbitrary shell commands
Streams stdout/stderr back in real-time chunks
Enables tool-use patterns inside a single long-running sandbox

🔭 Full Observability Stack

Layer	Technology	What's exposed
Metrics	Prometheus	`sandbox_startup_duration_seconds`, `active_sandbox_count`, container creation times
Tracing	OpenTelemetry (OTLP/HTTP)	Distributed spans across control plane → scheduler → node-agent
Dashboard	Vanilla HTML/JS	Real-time node fleet, sandbox table, leader election, eBPF events

Metrics are scraped from :9090/metrics on the control plane. The trace exporter is configured via OTEL_EXPORTER_OTLP_ENDPOINT.

Getting Started

Prerequisites

Tool	Min version	Purpose
Go	1.22	Control plane + scheduler
Rust	stable	Node agent
Docker / containerd	1.7	Container runtime
etcd	3.5	Cluster state
protoc	3.x	Proto compilation (dev only)
kubectl	1.29	Kubernetes deployment

Quick Start (local, no Kubernetes)

# 1. Clone
git clone https://github.com/runtime-fabric/fabric.git
cd fabric

# 2. Start etcd (Docker)
docker run -d --name etcd \
  -p 2379:2379 \
  quay.io/coreos/etcd:v3.5.12 \
  etcd --advertise-client-urls http://0.0.0.0:2379 \
       --listen-client-urls http://0.0.0.0:2379

# 3. Run the control plane
cd control-plane
go run ./cmd/server --etcd-endpoints=http://localhost:2379

# 4. Run the scheduler (separate terminal)
cd scheduler
go run ./cmd/scheduler --etcd-endpoints=http://localhost:2379

# 5. Run a node agent (separate terminal, requires containerd)
cd node-agent
cargo run --release -- --etcd-endpoint http://localhost:2379 \
                        --node-id node-alpha \
                        --listen-addr 0.0.0.0:50051

# 6. Open the dashboard
open dashboard/index.html

Running on Kubernetes

# Deploy with default Helm values
helm install fabric ./deploy/helm/fabric \
  --set etcd.endpoints=http://etcd:2379 \
  --set nodeAgent.runtimeClass=gvisor \
  --namespace fabric-system --create-namespace

# Watch the rollout
kubectl rollout status deployment/fabric-control-plane -n fabric-system

Configuration

Control Plane

Env var	Default	Description
`FABRIC_ETCD_ENDPOINTS`	`http://localhost:2379`	etcd cluster endpoints (comma-separated)
`FABRIC_LISTEN_ADDR`	`:8080`	gRPC + REST listen address
`FABRIC_METRICS_ADDR`	`:9090`	Prometheus metrics endpoint
`OTEL_EXPORTER_OTLP_ENDPOINT`	`""`	OTLP trace exporter URL (disabled if empty)
`FABRIC_WARM_POOL_SIZE`	`3`	Pre-warmed sandboxes per image
`FABRIC_CPU_OVERCOMMIT`	`1.5`	Maximum CPU overcommit ratio

Node Agent

Env var	Default	Description
`FABRIC_NODE_ID`	hostname	Unique node identifier registered in etcd
`FABRIC_ETCD_ENDPOINT`	`http://localhost:2379`	etcd endpoint
`FABRIC_GRPC_ADDR`	`0.0.0.0:50051`	gRPC listen address
`FABRIC_RUNTIME_CLASS`	`runsc`	OCI runtime class (`runc` or `runsc`)
`FABRIC_HEARTBEAT_TTL`	`15`	etcd lease TTL in seconds
`FABRIC_EBPF_ENABLED`	`true`	Enable eBPF syscall monitoring

API Reference

The primary API is defined in api/proto/v1/sandbox.proto.

Core RPCs

// Create and start a sandbox
rpc CreateSandbox(CreateSandboxRequest) returns (CreateSandboxResponse);

// Execute a command inside a running sandbox (streaming)
rpc ExecCommand(ExecCommandRequest) returns (stream ExecOutput);

// Retrieve stdout/stderr logs
rpc StreamLogs(StreamLogsRequest) returns (stream LogChunk);

// Terminate and clean up a sandbox
rpc TerminateSandbox(TerminateSandboxRequest) returns (TerminateSandboxResponse);

// Execute a multi-step task chain
rpc ExecuteChain(ExecuteChainRequest) returns (ExecuteChainResponse);

Quick example (grpcurl)

# Create a Python sandbox
grpcurl -plaintext -d '{
  "image": "python:3.11-slim",
  "tenant_id": "my-org",
  "resource_limits": { "cpu_millicores": 500, "memory_mb": 512 }
}' localhost:8080 fabric.v1.SandboxService/CreateSandbox

# Execute code inside it
grpcurl -plaintext -d '{
  "sandbox_id": "sb-cf9a3",
  "command": ["python", "-c", "print(42)"]
}' localhost:8080 fabric.v1.SandboxService/ExecCommand

Dashboard

A self-contained, zero-dependency observability UI lives at dashboard/index.html.

Open it in any modern browser — no server required. In production, serve it via any static file host (Nginx, Caddy, S3+CloudFront) and point it at the control plane's WebSocket/REST endpoint.

Panels included:

Cluster Overview — KPI strip (active sandboxes, avg startup, cluster CPU, security events)
etcd Leader Election — live leader/follower topology with TTL countdown
Node Fleet — per-node CPU/memory bars, sandbox dots, zone info
Live Event Feed — real-time audit trail from the control plane
Sandbox Table — full list with tenant, node, image, resource usage, status
eBPF Security Monitor — blocked syscall counts, threat severity list, heatmap
Prometheus Metrics — sparkline charts for startup latency, CPU, and memory

Project Structure

fabric/
├── api/
│   └── proto/v1/          # sandbox.proto (gRPC service definitions)
├── control-plane/
│   ├── cmd/server/        # main entry point
│   └── internal/
│       ├── api/           # gRPC + REST handlers
│       ├── chain/         # Task chain executor
│       ├── controller/    # Kubernetes controller (Sandbox CRD)
│       ├── metrics/       # Prometheus instrumentation
│       ├── pool/          # Warm sandbox pool manager
│       ├── state/         # etcd client, IPAM, store
│       ├── tracing/       # OpenTelemetry setup
│       └── webhook/       # Admission webhook
├── scheduler/
│   ├── cmd/scheduler/     # main entry point
│   └── internal/
│       ├── client/        # gRPC dispatcher → node agents
│       ├── registry/      # Node registry (etcd leases, heartbeats)
│       └── scheduler/     # Bin-packing engine
├── node-agent/
│   └── src/
│       ├── main.rs        # gRPC server, sandbox lifecycle
│       ├── metrics.rs     # Prometheus counters (Rust)
│       ├── registrar.rs   # etcd self-registration + heartbeat
│       ├── storage.rs     # Ephemeral scratch storage
│       ├── tools.rs       # Tool-use helper primitives
│       ├── network/       # CNI attachment, IPAM
│       ├── runtime/       # containerd / gVisor integration
│       └── security/      # eBPF + seccomp policy enforcement
├── cli/                   # `fabric` developer CLI
├── deploy/                # Kubernetes manifests, Helm chart
├── dashboard/
│   └── index.html         # Real-time cluster observability UI
├── pkg/                   # Shared Go packages
├── Makefile
├── go.mod
└── Cargo.toml

Development

# Lint + vet
make lint

# Run all Go tests
make test

# Regenerate protobuf
make proto

# Build all binaries
make build

# Build node agent (Rust)
make build-agent

# Run integration tests (requires etcd + containerd)
make test-integration

Running Tests

# Unit tests (Go)
go test ./...

# Unit tests (Rust)
cd node-agent && cargo test

# State store tests
go test ./control-plane/internal/state/... -v

Roadmap

Priority	Feature	Status
🔥 High	CRIU memory snapshotting (sub-50ms starts)	Planned
🔥 High	`fabric-cli` binary (`fabric run --image python`)	In Progress
🔶 Medium	Artifact upload API (S3/GCS auto-upload on completion)	Planned
🔶 Medium	Network isolation via Cilium CNI	Planned
🔵 Low	Persistent shared volumes (multi-sandbox RW)	Planned
🔵 Low	Fluent-bit log aggregation → Loki	Planned
🔵 Low	KubeVirt integration (VM-level isolation)	Research

Contributing

Contributions are welcome. For substantial changes, please open an issue first to discuss the approach.

Fork the repo
Create a feature branch (git checkout -b feat/my-feature)
Commit your changes with conventional commits (feat:, fix:, docs:)
Open a pull request against main