E2B Infrastructure Architecture

This document explains what this repository implements, what each service does, and how the services interact. It is the fastest way to build a mental model of the codebase before diving into the code.

Keep this document updated. If a code change alters anything described here (service responsibilities, ports, protocols, data stores, flows, deployment topology), update this file in the same PR.

What this repository implements

E2B provides sandboxes: isolated Linux VMs that start almost instantly (they resume pre-booted snapshots instead of cold-booting), run arbitrary code (typically generated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole backend: the control-plane REST API, the data-plane VM orchestration built on Firecracker microVMs, the in-VM agent, the edge routing layer, template building, and the Terraform/Nomad infrastructure to deploy it all on GCP (AWS in beta).

Two ideas drive the design:

1. A sandbox is a resumed snapshot. Templates are pre-booted VM snapshots (memory + disk + VM state) stored in object storage. "Creating" a sandbox means restoring a snapshot, which is why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root filesystem is a copy-on-write overlay, so only touched data is ever fetched.
2. Control plane and data plane are separate. The API decides where a sandbox runs and tracks that it runs (Postgres/Redis); the orchestrator on each node owns how it runs (Firecracker, networking, storage). Sandbox traffic never passes through the API.

System overview

flowchart TB
    subgraph clients["Clients"]
        SDK["SDK / CLI"]
        Browser["Browser / HTTP clients"]
        Docker["docker push"]
    end

    LB["Load balancer<br/>api.* | *.domain wildcard | docker.*"]

    VC["volume-content API (belt)<br/>api.&lt;domain&gt;"]

    subgraph controlplane["Control plane (API node pool)"]
        API["API<br/>REST :80, gRPC :5009/:5109"]
        DashAPI["dashboard-api :3010"]
        CP["client-proxy<br/>:3002"]
        DRP["docker-reverse-proxy :5000"]
    end

    subgraph datastores["State"]
        PG[("PostgreSQL<br/>teams, templates, builds, snapshots")]
        RD[("Redis<br/>running sandboxes, routing catalog, caches")]
        CH[("ClickHouse<br/>metrics, events, optional logs")]
        OS[("Object storage GCS/S3<br/>template + snapshot artifacts")]
    end

    subgraph clientnode["Sandbox nodes (one orchestrator per node)"]
        ORCH["orchestrator<br/>gRPC :5008, proxy :5007"]
        subgraph vm["Firecracker microVM (per sandbox)"]
            ENVD["envd :49983"]
            USERPROC["user processes"]
        end
    end

    subgraph buildnode["Build nodes"]
        TM["template-manager<br/>(orchestrator binary, gRPC :5008)"]
    end

    SDK -->|REST| LB --> API
    Browser -->|"port-sandboxid.domain"| LB --> CP
    Docker --> LB --> DRP
    API -.->|"mint content token + domain"| SDK
    SDK -->|"volume content (token-authed)<br/>api.&lt;BYOC or default domain&gt;"| VC
    API -->|"gRPC Create/Delete/Pause"| ORCH
    API -->|"gRPC TemplateCreate"| TM
    CP -->|"lookup sandbox → node"| RD
    CP -->|"forward :5007"| ORCH
    CP -.->|"gRPC auto-resume"| API
    ORCH --> ENVD
    ENVD --> USERPROC
    API --> PG & RD & CH
    DashAPI --> PG & CH
    ORCH --> OS & CH
    TM --> OS

Services

Service	Package	Runs on	Purpose
API	packages/api	API nodes	Public REST API; sandbox lifecycle, placement, auth, quotas
Orchestrator	packages/orchestrator	every sandbox node	Runs Firecracker VMs; sandbox create/pause/resume/kill
Template manager	packages/orchestrator (role)	build nodes	Builds templates from Docker images
Client proxy	packages/client-proxy	API nodes	Edge router: sandbox URL → correct node
Envd	packages/envd	inside every VM	In-VM agent: process/filesystem API for SDKs
Dashboard API	packages/dashboard-api	API nodes	Backend for the web dashboard (teams, builds, admin)
Docker reverse proxy	packages/docker-reverse-proxy	API nodes	Registry auth gateway for pushing template images

Supporting packages: packages/shared (protos, telemetry, storage clients, feature flags), packages/auth (authentication library), packages/db (Postgres migrations + sqlc queries), packages/clickhouse (ClickHouse schema + clients), packages/otel-collector (collector config), packages/nomad-nodepool-apm (autoscaler plugin), packages/local-dev (local stack).

API (packages/api)

The control-plane entry point (Gin, OpenAPI-generated from spec/openapi.yml, port 80).

• Resources: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs), templates and builds, teams, volumes, API keys/access tokens, admin operations.
• Auth (via packages/auth): team API keys (X-API-Key, e2b_ prefix), auth-provider JWTs (OIDC), admin token. Backed by an auth DB (Postgres) with a Redis team cache.
• Workload identity: sandbox create accepts an optional iam.tokens map of caller-named workload-token definitions (each an exact audience and tokenType). A non-empty, validated map enables workload identity, whose identity the orchestrator derives from the sandbox's already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the orchestrator in SandboxConfig.iam. The API mints no credential and delivers nothing into the sandbox; file-based delivery is rejected at admission. Definitions are persisted in the running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and orchestrator re-sync; a fork starts a new workload and does not inherit them.
• Placement: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a static list). Chooses a node per sandbox with a best-of-K algorithm (internal/orchestrator/placement/): sample K ready nodes, score by CPU commitment/usage, pick the lowest; retry on exhausted nodes. Tunable live via feature flags.
• State: writes sandbox records to Redis (source of truth for running sandboxes) and the sandbox→node routing catalog in Redis that client-proxy reads. Persistent entities (templates, builds, snapshots, teams) live in Postgres.
• Extra listeners: internal gRPC :5009 and edge gRPC :5109 expose ResumeSandbox so client-proxy can wake paused sandboxes on incoming traffic.
• Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to Loki, with a LaunchDarkly-gated ClickHouse read path for local-cluster logs during the log storage migration. LaunchDarkly feature flags also gate placement parameters, rate limits, and rollouts.

Orchestrator (packages/orchestrator)

A single Go binary running on every sandbox node (as root). ORCHESTRATOR_SERVICES selects its roles: orchestrator (run sandboxes) and/or template-manager (build templates). Code lives under pkg/, almost all Linux-only.

gRPC services on :5008 (pkg/server/, pkg/service/, pkg/template/server/, pkg/volumes/):

• SandboxService — Create, Update, List, Delete, Pause, Checkpoint.
• TemplateService — TemplateCreate, TemplateBuildStatus, TemplateBuildDelete (template-manager role only).
• InfoService — node identity, roles, capacity, health status (used by API node discovery).
• ChunkService / VolumeService — peer-to-peer template chunk serving; persistent volumes.

Key mechanisms (all under pkg/sandbox/):

• Firecracker (fc/): each sandbox is one Firecracker process in its own cgroup and network namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots. Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.
• Lazy memory / UFFD (uffd/): on resume, Firecracker restores the VM without loading memory; a userfaultfd handler serves page faults directly from the template's memfile, so only touched pages are read. An optional prefetcher warms known-hot pages.
• Copy-on-write rootfs (rootfs/, nbd/, block/): the template rootfs stays read-only; writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.
• Template cache (template/): templates are fetched lazily from object storage and cached on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other nodes before upload completes).
• Networking (network/): each sandbox gets a slot — a network namespace with a veth pair and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and reused; slot allocation is coordinated through Consul KV.
• Sandbox proxy (:5007, pkg/proxy/): reverse-proxies incoming traffic from client-proxy to the sandbox's slot IP and requested port, enforcing per-sandbox traffic access tokens.
• Writes sandbox lifecycle events and cgroup host stats to ClickHouse; exports metrics via OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy collector remains the fallback primary destination, and configured shadow destinations can mirror writes during collector/storage migrations without changing sandbox behavior.

Envd (packages/envd)

The agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.

• Process service (spec/process/process.proto): start/list/connect to processes, stream stdout/stderr, stdin, signals, PTYs — this is what SDKs use to "run code".
• Filesystem service (spec/filesystem/filesystem.proto): stat/list/make/move/remove/watch.
• REST: /health, /metrics, /files upload/download, /init (orchestrator pushes env vars, access token, metadata after boot/resume), /upgrade (live self-upgrade, below), freeze/thaw hooks used during pause.
• Auth: X-Access-Token header checked against a token delivered via Firecracker MMDS; signed URLs for file endpoints.
• Live upgrade (internal/services/process/upgrade.go): an authenticated POST /upgrade lets the orchestrator swap envd inside a running sandbox at resume. It streams the new binary in the request body and envd syscall.Execs into it with the same PID, carrying the workload's stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade /init restores the access token (so no re-adopted process runs unauthenticated), and the handover outcome (procs/watchers re-adopted, plus any failures) rides back on that /init's X-Envd-Handover header for fleet visibility.
• Scans guest ports and forwards them so any port a user process opens becomes reachable through sandbox URLs. pkg/version.go must be bumped on every behavioral change — the API and the orchestrator gate features on the envd version recorded in each template build.

Client proxy (packages/client-proxy)

The stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates https://<port>-<sandboxID>.<domain> requests (host parsing in packages/shared/pkg/proxy/host.go), looks the sandbox up in the Redis routing catalog to find the owning node, and reverse-proxies to that node's orchestrator proxy on :5007. If the sandbox is not in the catalog (paused), it calls the API's ResumeSandbox gRPC and retries — paused sandboxes wake transparently on traffic.

Dashboard API (packages/dashboard-api)

A separate REST service (port 3010, spec spec/openapi-dashboard.yml) consumed by the web dashboard, not the SDK: team management/provisioning, template tags, build listings, admin bootstrap. Team-scoped template and build read routes accept either dashboard user auth or team API key auth (X-API-Key). Its workspace-agnostic /v1/management operations are defined in the same dashboard OpenAPI contract and registered on the existing router. Their AdminJWTAuth OpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api /.well-known/jwks.json endpoint, with accepted signing methods derived from each JWK's required alg metadata. Issuers and audiences are configured through the JSON ADMIN_AUTH_PROVIDER_CONFIG value — the same config shape as AUTH_PROVIDER_CONFIG. Talks to Postgres and ClickHouse; never talks to orchestrators.

The /v1/management operations are the cluster's half of a contract the workspace residency owns: project upsert (a project is a public.teams row created from a caller-supplied UUID; the tier is assigned once at creation from a local default and no push moves it; a changed slug renames the project, and nothing else follows it), member sync (granular and batched, over opaque user UUIDs in users_teams), limit sync (into project_limits, which team_limits reads in preference to tiers), and user purge (memberships and access tokens; the public.users row survives). All are idempotent, because the caller is level-triggered and retries. Membership writes live in internal/management with their cache evictions rather than in the handlers: auth caches a copy of the team per member, and the sweep that would find those keys reads users_teams, so a removal has to name them itself.

DELETE /v1/management/projects/{teamID} is declared and answers 501. envs, snapshots and volumes reference teams with ON DELETE NO ACTION and templates are only soft-deleted, so a project that ever built one pins its team row — and releasing it needs the API service's orchestrator connections, which this service does not have. Projects are not deleted from control planes today.

Docker reverse proxy (packages/docker-reverse-proxy)

A Docker Registry v2 auth gateway (port 5000). Users docker push template base images with E2B credentials; the proxy validates them, swaps in real registry credentials, and rewrites paths into the cloud artifact registry (/v2/e2b/custom-envs/<templateID> → project registry).

Data stores

Store	Owner packages	What lives there
PostgreSQL	packages/db (goose migrations, sqlc)	Durable control-plane state: teams, users, tiers (quota defaults), project_limits (per-team quota overrides pushed in by the owning service; the team_limits view reads it in preference to tiers), envs (templates), env_builds (build rows: vcpu, ram_mb, status, versions), env_aliases, snapshots (paused sandboxes), team_api_keys, access_tokens, volumes, clusters
Redis	API, client-proxy, orchestrator	Ephemeral runtime state: running-sandbox store (source of truth), sandbox→node routing catalog, team/template/snapshot caches, rate limiting, P2P chunk peer registry
ClickHouse	packages/clickhouse	Time-series/analytics: metrics_gauge/metrics_sum (written by the OTel collector), sandbox_events, sandbox_host_stats (written by orchestrator), team metrics, and optionally sandbox_logs during the log migration. Read by API and dashboard-api
Object storage (GCS/S3/local, packages/shared/pkg/storage)	orchestrator, template-manager	Template & snapshot artifacts, keyed by build ID: {buildID}/memfile, {buildID}/rootfs.ext4, {buildID}/snapfile, {buildID}/metadata.json + .header index files
Consul KV	orchestrator	Network slot allocation across restarts

A template and a paused-sandbox snapshot have the same artifact shape — a snapshot is just a new build whose memfile/rootfs are stored as diffs against the template it came from (diff chains are resolved through the .header files).

Core flows

Sandbox creation

sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant R as Redis
    participant O as Orchestrator (chosen node)
    participant FC as Firecracker
    participant E as envd (in VM)

    C->>API: POST /sandboxes {templateID}
    API->>API: auth team, resolve template alias → ready build (Postgres/cache)
    API->>API: best-of-K placement → pick node
    API->>O: gRPC SandboxService.Create(SandboxConfig)
    O->>O: fetch template (local cache / NFS / object storage)
    O->>O: acquire network slot + NBD rootfs overlay + uffd memory
    O->>FC: load snapshot, resume VM
    O->>E: POST /init (env vars, access token) — retried until ready
    E-->>O: 204
    O-->>API: Create OK
    API->>R: store running sandbox + routing catalog entry
    API-->>C: 201 sandbox {sandboxID, domain}

The API blocks on the gRPC Create, which itself blocks on envd's /init — when the client gets a response, the sandbox is fully usable. Fresh creates are internally a resume of the template's base snapshot (cold boots only happen for filesystem-only templates and builds).

Sandbox traffic

sequenceDiagram
    autonumber
    participant U as Client
    participant CP as client-proxy :3002
    participant R as Redis catalog
    participant API as API
    participant OP as orchestrator proxy :5007
    participant E as envd / user process

    U->>CP: https://3000-i7fa3.domain
    CP->>CP: parse host → port 3000, sandbox i7fa3
    CP->>R: GetSandbox(i7fa3)
    alt running
        R-->>CP: node IP
    else paused / unknown
        CP->>API: gRPC ResumeSandbox(i7fa3)
        API-->>CP: node IP (after resume)
    end
    CP->>OP: forward to http://nodeIP:5007
    OP->>OP: lookup sandbox, check traffic access token
    OP->>E: http://slotIP:3000 (via veth/tap into VM)
    E-->>U: response

Volume content

Persistent volumes (packages/orchestrator/pkg/volumes/) are managed through the control-plane API (POST/GET /volumes), but their content — reading and writing files — is served by a separate volume-content API (belt, e2b-dev/belt) that the SDK talks to directly, not through the control-plane API. The API's role is to mint the credential and tell the SDK where to send content traffic.

sequenceDiagram
    autonumber
    participant U as SDK
    participant API as API
    participant PG as PostgreSQL
    participant VC as volume-content API (belt)

    U->>API: POST /volumes (create) or GET /volumes/{id}
    API->>PG: persist / load volume row
    API->>API: mint JWT (aud = https://api.&lt;domain&gt;)<br/>resolve domain
    API-->>U: { volumeID, name, token, domain? }
    Note over U: domain is returned only for BYOC teams;<br/>SDK stores it and falls back to api.&lt;E2B_DOMAIN&gt; otherwise
    U->>VC: /volumecontent/{id}/... at api.&lt;domain&gt;<br/>Authorization: Bearer token
    VC->>VC: verify token (audience must match its own origin)
    VC-->>U: file content

• Domain selection. The token's audience and the content host are the same origin, https://api.<domain>. For teams on a custom (BYOC) cluster (team.ClusterID set), the API returns that cluster's domain (cluster.SandboxDomain, resolved in handlers.volumeContentDomain) so content traffic goes to the BYOC cluster's edge instead of the control-plane host. For teams on the default cluster the response omits domain and the SDK uses its configured default (api.<E2B_DOMAIN>); the audience then uses the deployment's DOMAIN_NAME.
• Token. A short-lived JWT (handlers.generateVolumeContentToken, config in cfg.VolumesTokenConfig) signed by the API, scoped to the team and volume, presented as a bearer token on every content request. Its aud claim is https://api.<domain>, so a token minted for one cluster's origin is not accepted by another.

Pause and resume

• Pause: API records a snapshot row in Postgres, then gRPC Pause to the node. The orchestrator pauses the VM, snapshots it, diffs memory (dirty-page tracking) and rootfs (COW cache) against the template, caches the snapshot locally, and uploads asynchronously to object storage (with a retry budget). The sandbox leaves the Redis catalog.
  • Deferred rootfs export (gated by the deferred-rootfs-export flag in packages/shared/pkg/featureflags): instead of diffing the rootfs on the pause critical path, the orchestrator ejects the writable COW cache during pause and returns, then seals it into the rootfs diff (reflink) in the background. This moves the rootfs-diff latency off the pause, but the local snapshot's rootfs body isn't materialized until the seal finishes, so the async upload — and any origin-node resume/prefetch that reads the rootfs diff — waits on the seal. A seal failure is permanent (it never re-runs), so the upload fails fast rather than retrying.
• Resume: same path as creation, but placement prefers the origin node — if the snapshot is still in its local cache, resume avoids any object-storage reads. Checkpoint is a pause+resume in place used to persist state while keeping the sandbox running.
• Envd live-upgrade on resume: the orchestrator can upgrade the sandbox's envd to a newer node-local build during resume (gated by the envd-upgrade-target flag in packages/shared/pkg/featureflags), via envd's POST /upgrade (see the envd section). It is best-effort — a delivery failure before the exec leaves the old envd serving — except an unrecoverable post-exec failure (the new envd never re-initializes), which fails the resume rather than return a permanently unusable sandbox.
• Auto-pause/auto-resume make sandboxes effectively serverless: idle sandboxes pause, traffic resumes them (see traffic flow above).

Template build

sequenceDiagram
    autonumber
    participant C as SDK / docker push
    participant DRP as docker-reverse-proxy
    participant API as API
    participant TM as template-manager (build node)
    participant FC as Firecracker build VMs
    participant OS as Object storage

    opt custom base image
        C->>DRP: docker push (E2B token)
        DRP->>DRP: swap credentials, rewrite path → artifact registry
    end
    C->>API: POST /v3/templates (register build: cpu, ram) → Postgres env_builds
    C->>API: POST /v2/templates/{id}/builds/{buildID} (recipe: steps, start/ready cmd)
    API->>TM: gRPC TemplateCreate(TemplateConfig)
    TM->>TM: pull image → inject envd/provisioning → extract ext4 rootfs
    TM->>FC: boot VM per phase: provision → user steps
    TM->>TM: resize disk on host
    TM->>FC: boot VM per phase: finalize → optimize
    TM->>OS: upload layers + final {buildID}/memfile, rootfs.ext4, snapfile, metadata
    API->>TM: poll TemplateBuildStatus
    API->>API: mark build ready in Postgres

Builds are layered (pkg/template/build/phases/): base → user → one layer per recipe step → resize disk → finalize → optimize. Each layer is hashed and cached, so rebuilds only re-run changed steps. Resize disk grows the quiescent rootfs on the host; the other non-cached phases run in a real Firecracker VM and their pause-diffs become layers. The optimize phase records which memory pages a fresh resume touches, producing prefetch hints that speed up future sandbox starts.

Deployment topology

Deployed with Terraform (iac/provider-gcp/, iac/provider-aws/) onto a Nomad + Consul cluster. Nomad job specs live in iac/modules/job-*/jobs/*.hcl.

flowchart TB
    LB["Cloud load balancer + TLS<br/>api.* → API | *.domain → client-proxy | docker.* → registry proxy"]

    subgraph servers["server pool (3 nodes)"]
        NS["Nomad + Consul servers (control plane)"]
    end
    subgraph apipool["api pool"]
        AJ["api, dashboard-api, client-proxy,<br/>ingress (Traefik), docker-reverse-proxy,<br/>redis, loki, otel-collector, autoscaler"]
    end
    subgraph clientpool["default pool (autoscaled)"]
        OJ["orchestrator (system job, raw_exec)<br/>+ Firecracker sandboxes"]
    end
    subgraph buildpool["build pool (autoscaled)"]
        TJ["template-manager (raw_exec)"]
    end
    subgraph chpool["clickhouse pool"]
        CJ["clickhouse + backups"]
    end

    LB --> apipool
    AJ -->|gRPC| OJ & TJ
    NS -.->|schedules jobs| apipool & clientpool & buildpool & chpool

• Server nodes run only Nomad/Consul servers (scheduling, service discovery, Consul DNS — services address each other as *.service.consul).
• API nodes host every control-plane container and are the only LB backend.
• Sandbox ("client") nodes run the orchestrator as a Nomad system job via raw_exec (it needs root for Firecracker, namespaces, NBD, cgroups). Configured with hugepages and local template caches. Autoscaled.
• Build nodes run the same binary in template-manager mode; the nomad-nodepool-apm autoscaler plugin scales the job with the node pool.
• PostgreSQL is external (connection string via secrets); Redis runs as a Nomad job or as a managed service; ClickHouse runs on its own pool.
• Observability: everything exports OTel; the collector fans out to ClickHouse (product metrics) and Grafana Cloud/stack. Logs default to the legacy Vector → Loki path; dynamic log routing can select a primary collector and shadow collectors, and local-cluster log reads can be switched to ClickHouse with logs-read-config after sandbox_logs is populated.

Repository layout

packages/
  api/                  Control-plane REST API
  orchestrator/         Sandbox runtime + template builder (one binary, per-node)
  client-proxy/         Edge router for sandbox traffic
  envd/                 In-VM agent (bump pkg/version.go on behavior change!)
  dashboard-api/        Web-dashboard backend
  docker-reverse-proxy/ Registry auth gateway for template images
  shared/               Protos, telemetry, storage clients, proxy engine, feature flags
  auth/                 AuthN library (API keys, JWT/OIDC) used by api + dashboard-api
  db/                   Postgres migrations (goose) + queries (sqlc)
  clickhouse/           ClickHouse schema, batching writers, query clients
  otel-collector/       Collector config
  nomad-nodepool-apm/   Nomad autoscaler metric and deployment-aware target plugins
  local-dev/            docker-compose local stack + DB seeding
spec/                   OpenAPI specs (public, edge, dashboard) — codegen sources
iac/                    Terraform + Nomad jobs (provider-gcp, provider-aws, shared modules)
tests/integration/      Integration tests against a live deployment

Cross-service contracts are all generated: OpenAPI specs in spec/, gRPC protos in packages/orchestrator/*.proto and packages/envd/spec/, SQL in packages/db/queries/. Run make generate after changing any of them.