SIE Gateway (Rust)

August 27, 2026 · View on GitHub

Runtime gateway for elastic GPU inference deployments. It routes the four SIE primitives over NATS JetStream: ordinary encode, score, and extract requests use pool queues (at-least-once delivery), while capped logical batch pools may target an assigned worker directly. OpenAI-compatible and SIE-native generation endpoints (/v1/chat/completions, /v1/completions, /v1/responses, /v1/generate/{model}) use per-worker direct-dispatch subjects with cache-aware prefix routing. The gateway also owns pool coordination and worker health. Config writes live in sie-config; the gateway is a pure consumer of config state, bootstrapping from GET /v1/configs/export and subscribing to NATS deltas.

Generation is treated as a supported fourth primitive.

See docs/architecture-guide.md for the authoritative code-audited architecture document (covers both this service and sie-config).

Features

Two dispatch modes — pool-subject JetStream for ordinary encode/score/extract; per-worker JetStream direct-dispatch for capped logical batch pools and the four generation endpoints (cache-aware prefix routing, pool fallback, first-chunk timeout fallback)
Worker discovery — static URLs or Kubernetes service endpoints
Health monitoring — WebSocket streaming or NATS heartbeats
Queue transport — JetStream work publish, reply inbox collection, backpressure handling, and DLQ republish
Pool management — named pools with TTLs, minimum worker counts, and Kubernetes-backed coordination
Model registry — filesystem seed + background-retried snapshot from sie-config (GET /v1/configs/export) + live NATS deltas + periodic GET /v1/configs/epoch drift detection; the gateway is read-only
Config write cutover — POST /v1/configs/models is not registered on the gateway. Requests receive 405 Method Not Allowed from axum's default router. Writes belong to the control plane at SIE_CONFIG_SERVICE_URL
Worker-ack readiness — GET /v1/configs/models/{id}/status reports per-replica bundle_config_hash acknowledgement plus the local config_epoch for admin tooling polling after a sie-config write
Config distribution — authoritative deltas arrive on sie.config.models.* from sie-config; the gateway applies them to its in-memory registry
Auth — static-token auth via SIE_AUTH_TOKEN[S]; config write idempotency belongs to sie-config
Demand tracking and readiness — provisioning responses, pending-demand metrics, and worker ack checks after config changes
Observability — canonical OpenTelemetry metrics, privacy-safe logs, distributed tracing, audit middleware, and HTML/WebSocket status surfaces
Optional cloud storage — cloud-storage enables S3/GCS/Azure Blob and native Alibaba OSS V4 payload backends; the Docker build enables it

Quick Start

Requirements: Rust stable, NATS/JetStream for inference routing

# Build from the repo root (preferred contributor flow)
mise run gateway-build -- -r

# Direct cargo equivalent from this package directory
cargo build --release

# Run with static workers
SIE_NATS_URL=nats://localhost:4222 \
./target/release/sie-gateway serve \
  -w http://worker1:8080 \
  -w http://worker2:8080

# Run with Kubernetes discovery
SIE_NATS_URL=nats://localhost:4222 \
./target/release/sie-gateway serve \
  --kubernetes \
  --k8s-namespace sie \
  --k8s-service sie-worker

Notes:

encode / score / extract are JetStream-only: ordinary requests use pool queues, while capped logical batch pools may use worker-direct subjects. If no usable NATS client is available, these requests return 503.
Generation endpoints require per-worker JetStream direct-dispatch. NATS Core remains responsible for worker health, config distribution, and result delivery. There is no direct worker HTTP fallback mode and no SIE_CLUSTER_ROUTING toggle.

CLI

sie-gateway serve [OPTIONS]

Key options:
  -p, --port <PORT>              Listen port (default: 8080)
      --host <HOST>              Listen host (default: 0.0.0.0)
  -w, --worker <WORKERS>         Worker URL(s), repeatable
      --kubernetes               Enable Kubernetes discovery
      --k8s-namespace <NS>       K8s namespace (default: default)
      --k8s-service <SVC>        K8s service name (default: sie-worker)
      --k8s-port <PORT>          K8s worker port (default: 8080)
  -l, --log-level <LEVEL>        Log level (default: info)
      --json-logs                Enable structured JSON logging
      --health-mode <MODE>       Worker health transport (supported: ws, nats; default: ws)
      --bundles-dir <PATH>       Bundles directory
      --models-dir <PATH>        Models directory

sie-gateway version
sie-gateway openapi --output packages/sie_gateway/openapi.json

Each --flag above has a matching SIE_* environment variable (see next section); CLI flags override env vars.

Common Environment Variables

Variable	Default	Description
`SIE_GATEWAY_PORT`	`8080`	Listen port
`SIE_GATEWAY_HOST`	`0.0.0.0`	Listen host
`SIE_GATEWAY_WORKERS`		CSV of worker URLs
`SIE_GATEWAY_KUBERNETES`	`false`	Enable Kubernetes discovery
`SIE_GATEWAY_K8S_NAMESPACE`	`default`	K8s namespace
`SIE_GATEWAY_K8S_SERVICE`	`sie-worker`	K8s service name
`SIE_GATEWAY_K8S_PORT`	`8080`	K8s worker port
`SIE_GATEWAY_HEALTH_MODE`	`ws`	Health mode: `ws` or `nats`
`SIE_NATS_URL`		NATS server URL. The process can start without it, but inference requests will return `503` until a usable client exists
`SIE_AUTH_MODE`	`none`	Auth mode for inbound requests: `none` disables, `token` (alias `static`) enforces. Unknown values fail-open-to-bypass; `main` logs a startup error naming the bad value
`SIE_AUTH_TOKENS`		CSV of valid bearer tokens for inference and pool/config read endpoints. If unset, the singular `SIE_AUTH_TOKEN` is used as a fallback. When auth is enabled and this list is empty, non-probe requests return `500`
`SIE_AUTH_TOKEN`		Singular alias for `SIE_AUTH_TOKENS` (fallback only; prefer the plural form)
`SIE_ADMIN_TOKEN`		Admin bearer token the gateway (1) presents as a client to `sie-config` on `GET /v1/configs/export` and `GET /v1/configs/epoch`, and (2) requires inbound for admin-gated mutations: `POST/PUT/DELETE` on `/v1/configs/`, `/v1/admin/`, `/v1/pools/*`. If empty and an inbound request targets one of those paths, the middleware fails closed with `403`
`SIE_AUTH_EXEMPT_OPERATIONAL`	`false`	When `true`, `/`, `/health`, and `/ws/*` are exempt from auth (they expose worker URLs, queue depth, GPU inventory). `/healthz` and `/readyz` are always exempt (K8s probes carry no creds). Default is fail-closed
`SIE_NATS_CONFIG_TRUSTED_PRODUCERS`	`sie-config`	CSV allowlist of `producer_id` values trusted to publish on `sie.config.models._all`. Matches exact OR K8s pod-name prefix (`sie-config` also matches `sie-config-5f7b6d8c-kxwvr`). Untrusted notifications are dropped; the epoch poller still closes the gap
`SIE_NATS_CONFIG_TRUST_ANY_PRODUCER`	`false`	Disable producer validation entirely (dev/local only). `main` emits a startup audit warning when on
`SIE_LOG_LEVEL`	`info`	Log level (`debug`, `info`, `warn`, `error`)
`SIE_LOG_JSON`	`false`	Structured JSON logging (for Loki)
`SIE_GATEWAY_REQUEST_TIMEOUT`	`120.0`	Non-generation queue result wait timeout in seconds
`SIE_GATEWAY_MAX_STREAM_PENDING`	`50000`	Max pending stream items per JetStream work stream. Pool-wide: a stream is per pool, so this number cannot tell a hot model's backlog from a cold one's
`SIE_GATEWAY_MAX_LANE_IN_FLIGHT_ITEMS`	`10000`	Per-lane (`pool`/`machine_profile`/`bundle`) in-flight work-item ceiling. Always evaluated and recorded on `sie.gateway.queue.lane_admission.decisions`; only sheds when `SIE_GATEWAY_LANE_BACKPRESSURE_ENFORCE` is on
`SIE_GATEWAY_LANE_BACKPRESSURE_ENFORCE`	`false`	Act on the per-lane decision. Off = shadow mode: the decision is computed and recorded, and admission is governed by `SIE_GATEWAY_MAX_STREAM_PENDING` alone. On = a saturated lane gets a targeted 503 while other lanes on the same pool keep admitting
`SIE_GATEWAY_DEFAULT_MAX_TOKENS`	`1024`	Output-token cap applied to `/v1/chat/completions` requests that omit both `max_completion_tokens` and `max_tokens`. OpenAI treats the field as optional, so the gateway defaults rather than rejecting — generic clients (Open WebUI) rely on this
`SIE_GATEWAY_ENABLE_POOLS`	`false`	Enable pool management
`SIE_GATEWAY_HOT_RELOAD`	`false`	Enable filesystem watcher for bundle/model directories
`SIE_GATEWAY_WATCH_POLLING`	`false`	Use polling file-watcher instead of inotify/fsevents (alias: `SIE_GATEWAY_POLLING_WATCHER`). Useful on filesystems where native notifications are unreliable
`SIE_CONFIG_SERVICE_URL`	unset	Base URL of `sie-config`. When set, the gateway runs a background `GET /v1/configs/export` bootstrap on startup and a 30 s `GET /v1/configs/epoch` drift poller. When unset, the bootstrap/poller tasks no-op and the gateway runs filesystem-seed-only
`SIE_MULTI_ROUTER`	`false`	Multi-gateway coordination flag (wire-compatible name retained)
`SIE_GATEWAY_CONFIGURED_GPUS`		CSV of canonical machine profiles used for validation and default pool display
`SIE_GATEWAY_CONFIGURED_PHYSICAL_LANES`	`[]`	JSON array of exact queue/KEDA lanes, for example `[{"pool":"default","machineProfile":"cpu","bundle":"default"}]`. Queue routing fails closed when its resolved tuple is absent. Helm and the managed Modal gateway derive this catalog from their deployment manifests; standalone queue deployments must set it explicitly
`SIE_GATEWAY_GPU_ALIASES`		JSON map of request aliases to canonical machine profiles
`SIE_BUNDLES_DIR`	`bundles`	Optional bundle filesystem seed. Unset in default Helm deploys: the gateway pulls bundles from `sie-config` via `GET /v1/configs/bundles{,/{id}}` at startup and the registry's filesystem reload is a no-op. Only set by the `gateway.embeddedConfigs` / `gateway.configMap` overlays which mount a ConfigMap at `/configs/bundles`.
`SIE_MODELS_DIR`	`models`	Optional model filesystem seed. Same semantics as `SIE_BUNDLES_DIR`: unset in default deploys, runtime model writes always go to `sie-config` and the gateway replays them via `GET /v1/configs/export`.
`SIE_PAYLOAD_STORE_URL`	unset	Shared payload offload store path. When unset, large-payload offload is disabled. Queue deployments should use `s3://`, `gs://`, `abfs://`, `abfss://`, or `oss://`; local filesystem paths only work when gateway and workers share the same path
`SIE_OSS_REGION`	unset	Required explicit Alibaba region for `oss://`; OSS V4 is region-scoped
`SIE_OSS_USE_INTERNAL_ENDPOINT`	`false`	Derive the Alibaba VPC-internal HTTPS endpoint for `oss://`; arbitrary endpoint overrides are not accepted

API Endpoints

Health And Operator Surface

Method	Path	Description
GET	`/`	HTML status page
GET	`/healthz`	Liveness — `200`, `text/plain` body `ok`
GET	`/readyz`	Readiness — `200` + `ok` once the gateway process is serving (`text/plain`); worker availability is exposed by `/health`
GET	`/health`	Cluster health JSON
GET	`/openapi.json`	OpenAPI 3 contract for gateway-owned HTTP routes
GET	`/ws/cluster-status`	WebSocket cluster status feed
GET	`/v1/models`	List available models

Inference (encode / score / extract — JetStream queue)

Method	Path	Description
POST	`/v1/encode/{*model}`	Queue an encode request
POST	`/v1/score/{*model}`	Queue a score request
POST	`/v1/extract/{*model}`	Queue an extract request

Generation (direct-dispatch per-worker streams)

Generation is a supported fourth primitive. Strict allow-list parsing on the OpenAI-compatible endpoints means unknown fields reject with 400 unsupported_field.

Method	Path	Description
POST	`/v1/chat/completions`	OpenAI Chat Completions (streaming + non-streaming, `n`, `best_of`, `tools`, `response_format`, `lora_adapter`)
POST	`/v1/completions`	OpenAI legacy Completions (raw `prompt`, streaming + non-streaming, single-candidate)
POST	`/v1/responses`	OpenAI Responses API MVP (stateless single-turn, non-streaming)
POST	`/v1/generate/{*model}`	SIE-native generate (full `GenerateParams` envelope, streaming + non-streaming)

Common headers:

X-SIE-MACHINE-PROFILE
X-SIE-POOL
X-SIE-SDK-Version

Common behaviors:

404 for unknown models once the in-memory registry has bootstrapped from sie-config (fast-fail; avoids queueing requests for typo'd model ids)
503 + Retry-After + X-SIE-Error-Code: PROVISIONING on scale-from-zero, whether or not X-SIE-MACHINE-PROFILE was set (records pending demand for KEDA)
503 + Retry-After for no-consumer or backpressure publish failures
504 + X-SIE-Error-Code: GATEWAY_TIMEOUT + Retry-After: 5 on non-generation queue routes when a queued request was published but no worker result reached the gateway before SIE_GATEWAY_REQUEST_TIMEOUT. Generation routes use their own streaming/direct-dispatch timeout contract. Worker-emitted MODEL_LOADING remains a separate retryable 503 MODEL_LOADING signal.
503 + X-SIE-Error-Code: RESOURCE_EXHAUSTED + Retry-After: 5 when every item in a batch fails with the same retryable code (RESOURCE_EXHAUSTED from worker-side OOM recovery exhaustion, MODEL_LOADING from a worker still warming up). The SDK auto-retries with bounded exponential backoff. Mixed batches keep returning 500 all_items_failed with per-item code fields in the response body so callers can see which items hit which failure mode.

The same gateway-owned OpenAPI contract is available at runtime via GET /openapi.json and as a committed static artifact at packages/sie_gateway/openapi.json. Regenerate it with mise run openapi before committing API-surface changes.

Pool Management

Method	Path	Description
GET	`/v1/pools`	List pools
POST	`/v1/pools`	Create pool
GET	`/v1/pools/{name}`	Get pool details
POST	`/v1/pools/{name}/renew`	Renew pool TTL
DELETE	`/v1/pools/{name}`	Delete pool (default pool protected)

Config API (read-only)

The gateway is a pure consumer of config state. Writes live in sie-config; the gateway returns 405 Method Not Allowed for POST /v1/configs/models.

Method	Path	Description
GET	`/v1/configs/models`	List model configs known to this replica
GET	`/v1/configs/models/{*id}`	Get model config YAML, or — when the path ends in `/status` and the prefix matches a known model — a worker-ack readiness JSON document for admin tooling
GET	`/v1/configs/bundles`	List bundle configs
GET	`/v1/configs/bundles/{id}`	Get bundle config
POST	`/v1/configs/resolve`	Resolve a bundle for a model

GET /v1/configs/models/{id}/status is gateway-only (it reports per-replica worker-ack state from the gateway's in-memory WorkerRegistry). sie-config does not serve it.

Not registered on the gateway:

POST /v1/configs/models — write, owned by sie-config.
GET /v1/configs/export, GET /v1/configs/epoch — served by sie-config; the gateway is a client of both (bootstrap + drift poll). See packages/sie_gateway/docs/architecture-guide.md §4.

Docker

Build the gateway image from the repository root; the Helm chart and values are under deploy/helm/sie-cluster.

docker build -f packages/sie_gateway/Dockerfile -t sie-gateway .

Testing

# Preferred repo-root contributor flow
mise run gateway-fmt              # applies rustfmt (default); add `-- --check` for CI-style check-only
mise run gateway-test
mise run gateway-clippy

# Direct cargo equivalents from this package directory
cargo fmt --all                   # append `--check` to verify without writing
cargo test
cargo clippy --all-targets -- -D warnings

Project Structure

src/
  main.rs                CLI parsing and async runtime startup
  server.rs              Axum routes and AppState
  config.rs              Config loading from env/CLI
  error.rs               AppError -> HTTP status mapping
  metrics.rs             Compatibility aliases for canonical telemetry types
  observability/metrics.rs  Canonical OpenTelemetry metric facade
  handlers/
    health.rs            Health and status endpoints
    models.rs            GET /v1/models helpers
    pools.rs             Pool CRUD
    proxy.rs             Two dispatch modes: ordinary encode/score/extract pool queues + capped-pool/generation worker-direct
    config_api.rs        Read-only config API (GET /v1/configs/*, including /status dispatch); POST /v1/configs/models is NOT registered (gateway returns 405)
  middleware/
    auth.rs              Token authentication
    audit.rs             Request/response audit logging
  discovery/
    static_discovery.rs  Static worker list
    ws_health.rs         WebSocket worker health
    nats_health.rs       NATS worker health
    k8s_discovery.rs     Kubernetes endpoint discovery
  state/
    worker_registry.rs   Worker tracking and queue-pool resolution
    model_registry.rs    Model and bundle registry (in-memory)
    pool_manager.rs      Pool management
    k8s_pool_backend.rs  K8s ConfigMap/Lease pool storage
    k8s_pool_watcher.rs  K8s pool state watcher
    config_watcher.rs    Filesystem hot reload
    config_bootstrap.rs  Cold-start snapshot fetch from sie-config
    config_poller.rs     30 s epoch drift detector against sie-config
    config_epoch.rs      Monotonic config-epoch counter (AtomicU64 with CAS)
    demand_tracker.rs    Pending-demand tracking
  nats/
    manager.rs           NATS connection and config-delta subscription
  queue/
    publisher.rs         JetStream work publishing
    consumer.rs          Work consumption helpers
    dlq.rs               Dead-letter queue handling
    payload_store.rs     Payload offload storage

Key Dependencies

axum — HTTP framework
tokio — async runtime
clap — CLI parsing
async-nats — NATS/JetStream client
kube — Kubernetes client
OpenTelemetry — canonical OTLP telemetry
serde, serde_json, serde_yaml — config and payload serialization