English | 中文

	Ferro Labs AI Gateway
	Open-Source, OpenAI-Compatible LLM Gateway

High-performance AI gateway in Go. Route LLM requests across 30 providers via a single OpenAI-compatible API.

📖 Documentation: docs.ferrolabs.ai

🔀 30 providers, 2,500+ models — one API
⚡ 13,925 RPS at 1,000 concurrent users (v1.0.0 benchmark)
📦 Single static binary, no external services required, 32 MB base memory

Quick Start

Under two minutes from install to first response.

Then go from nothing to a served request:

export OPENAI_API_KEY=sk-your-key     # ferrogw init detects this and writes the matching target
ferrogw init                          # writes config.yaml, prints your master key
export GATEWAY_CONFIG=./config.yaml   # the server reads a config file only when this is set
export MASTER_KEY=fgw_your-master-key # the key ferrogw init printed
ferrogw serve                         # starts the server on :8080

ferrogw init prints the master key once and never writes it to disk — save it yourself, in your .env file or a secret manager.

Docker runs the server itself, so there is no ferrogw init to print a master key — choose your own. Pass both variables by name so their values stay off the command line, where ps would show them:

export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw-any-strong-secret-you-choose
docker run -p 8080:8080 -e OPENAI_API_KEY -e MASTER_KEY ghcr.io/ferro-labs/ai-gateway:latest

Passing them by name keeps the values out of process arguments only — they are still readable in the container's environment and through docker inspect, so use Docker secrets or your platform's secret storage in production.

First request

export MASTER_KEY=fgw_your-master-key   # the key ferrogw init printed, in whichever shell you curl from

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]
  }' | jq

Every release is signed with keyless cosign and ships an SPDX SBOM — verification steps are in SECURITY.md.

Why Ferro Labs AI Gateway

AI gateways differ most under load, in throughput, latency and memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput — a single binary that routes LLM requests with predictable latency and minimal resource usage.

Features

Dashboard

Every release binary serves a built-in operations console at / — same port as the API, compiled in with go:embed, no second image and no second origin. Sign in with your MASTER_KEY or any admin / read-only key and it reads the live gateway: traffic, spend, provider health, routing, plugins, request logs, and the audit trail.

The bundle is produced by the release pipeline, so a binary built with go install is the one exception: it answers / with a placeholder instead. Any other install method above ships the console.

Overview, Analytics (latency/TTFT/cost percentiles), Providers, Routing, Plugins, a Playground over the real routing path, Tracing, Request Logs, the Audit Trail, Configuration with history/rollback, and scoped API key management.

Run the gateway and open http://localhost:8080. To see it filled like the recording above, bring up the self-contained demo stack:

make up-fullstack   # gateway + Postgres + Jaeger + Prometheus + Grafana + mock upstream + load generator
# then open http://localhost:8080

Build and development details are in web/README.md.

Documentation

The root README is the overview; each subsystem keeps its own reference beside its code:

Examples

Integration examples for common use cases are in ferro-labs/ai-gateway-examples:

Configuration

One YAML/JSON file, named by GATEWAY_CONFIG, drives routing, guardrails, MCP tools, and observability:

strategy:
  mode: fallback              # 8 modes — see internal/strategies/README.md

targets:
  - virtual_key: openai       # an allowlist: only listed providers are routable
    retry: { attempts: 3 }    # per target, honoured under every routing mode
    concurrency: { max_concurrency: 32, queue_size: 1000 }
  - virtual_key: anthropic

aliases:
  fast: gpt-4o-mini

plugins:                      # guardrails first, then cache — see plugin/README.md
  - name: word-filter
    type: guardrail
    stage: before_request
    enabled: true
    config: { blocked_words: ["password", "secret"] }

mcp_servers:                  # tool servers — see mcp/README.md
  - name: search
    url: https://mcp.example.com/mcp
    headers: { Authorization: "Bearer ${SEARCH_TOKEN}" }

${VAR} references (braced form only) resolve when a component is constructed, never at file load — so secrets are never stored, served by GET /admin/config, or restored by a rollback. A bare $ is data; an undefined variable is an error.

The full annotated reference with every option is config.example.yaml / config.example.json, and the schema guide is config/README.md. ferrogw validate checks a file without starting the server.

Key environment variables

See AGENTS.md for the full environment variable reference including provider API keys, store backends, and OTel settings.

Observability

See everything your gateway does — every request, what it cost, how long it took, which provider served it, and which guardrails ran. Ferro Labs AI Gateway ships first-class OpenTelemetry tracing and Prometheus metrics out of the box, and stays at a zero-allocation no-op until you turn it on. Point it at Jaeger, Grafana, New Relic, LangSmith, Datadog, or Honeycomb — anything that speaks OTLP — and every request emits a gateway.request span carrying GenAI semantic conventions (gen_ai.*) plus ferro.* extensions for cost, routing, MCP tool calls, and stream timings. The same trace ID threads your logs, spans, and the X-Request-ID response header.

📈 Full observability guide → observability/README.md — managed-backend setup, endpoint & transport rules, every emitted attribute, privacy levels, and exporter plugins.

Bring up the gateway wired to a full monitoring stack — Prometheus, Grafana, and Jaeger — driven by generated traffic, in one command:

make up-fullstack   # then open Grafana at http://localhost:3000

Grafana — request rate, latency percentiles, per-provider breakdown, token cost, and circuit-breaker state, all from the gateway's Prometheus metrics.

Jaeger — one request's gateway.request span, opened to reveal its gen_ai.* and ferro.* attributes.

Enable tracing with one variable (or the observability: config block — endpoint, protocol, sampling, privacy, headers are all documented in the guide):

export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
ferrogw serve

CLI

ferrogw is a single binary — no separate CLI tool required.

Global flags available on all subcommands: --gateway-url, --api-key, --format (table/json/yaml).

Deployment

Local development

export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw_your-master-key
export GATEWAY_CONFIG=./config.yaml
make build && ./bin/ferrogw

Railway & Render

The deploy buttons at the top of this README provision either platform: Railway with SQLite on a volume (point API_KEY_STORE_DSN, CONFIG_STORE_DSN and REQUEST_LOG_STORE_DSN at paths under /data) or PostgreSQL, and Render from the repo's render.yaml Blueprint, which generates MASTER_KEY and wires the store DSNs to a managed Postgres automatically.

Docker Compose

Three Compose files in deploy/ follow the standard override pattern — a shared base, a dev override that builds from source, and a prod override with a pinned tag, health check, and resource limits. Run everything from the repository root:

make up             # dev: builds from source
IMAGE_TAG=v1.4.5 CORS_ORIGINS=https://your-domain.com make up-prod
make down           # tears down either

One container serves both the API and the dashboard — no second image, no second origin. Provider keys go in a repository-root .env or the environment. deploy/README.md has the full reference, including a self-contained PostgreSQL pairing and the fullstack observability stack.

Kubernetes via Helm

helm repo add ferro-labs https://ferro-labs.github.io/helm-charts
helm repo update
helm install ferro-gw ferro-labs/ai-gateway \
  --set env.OPENAI_API_KEY=sk-your-key

Helm charts: github.com/ferro-labs/helm-charts | ArtifactHub

Migrate to Ferro Labs AI Gateway

The gateway is OpenAI-compatible, so for a client already using an OpenAI-compatible SDK, migration — from another gateway or from calling a provider directly — is a base_url and api_key change: point the SDK at the gateway and present a gateway-issued credential in place of the provider's own. A client with its own API, such as LiteLLM's completion() below, also moves to the OpenAI SDK.

From LiteLLM

Before (LiteLLM):

from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

After (Ferro Labs AI Gateway):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Provider API keys move to environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, …); the model list becomes targets + aliases in config.yaml.

Why migrate from LiteLLM:

• 14x higher throughput at 150 concurrent users (2,447 vs 175 RPS)
• 23x less memory at peak load (47 MB vs 1,124 MB under streaming)
• Single binary — no Python environment, no pip, no virtualenv
• Predictable latency — p99 stays under 65 ms at 150 VU vs LiteLLM's timeouts at the same concurrency

From Portkey

The code change is the same one line — Ferro Labs uses the standard OpenAI SDK with no custom headers in self-hosted mode.

Why migrate from Portkey:

• Fully open source — no per-request pricing, no log limits
• Self-hosted — the gateway runs in your infrastructure, and prompts reach only the providers you configure
• No vendor lock-in — Apache 2.0 license
• MCP support — Portkey self-hosted lacks native MCP
• FerroCloud (coming soon) for teams that want a managed service

Performance

Every figure in this section comes from one run: Ferro Labs v1.0.0, measured 2026-03-23. Benchmarked against Kong OSS, Bifrost, LiteLLM, and Portkey on GCP n2-standard-8 (8 vCPU, 32 GB RAM) using a 60ms fixed-latency mock upstream — results reflect gateway overhead only. Later releases have not been re-measured; reproduce against the version you intend to run using the commands below.

At 1,000 VU: 13,925 RPS, p50 overhead 8.1ms, memory 135 MB. Against the live OpenAI API, the gateway itself adds 25 microseconds p50 in a typical plugin configuration, and 2 microseconds with no plugins enabled.

Full methodology, raw results, and flamegraph analysis: ferro-labs/ai-gateway-performance-benchmarks (make setup && make bench reproduces it).

FerroCloud

FerroCloud — the managed version of Ferro Labs AI Gateway with multi-tenancy, analytics, and cost governance — is coming soon.

👉 Join the waitlist at ferrolabs.ai

SDKs

Official client libraries for the Ferro Labs AI Gateway — and the standard OpenAI SDK works unchanged: point base_url at http://your-gateway:8080/v1.

from ferrolabs import FerroClient

client = FerroClient(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

import { FerroClient } from "ferrolabs";

const client = new FerroClient({
  baseURL: "http://localhost:8080/v1",
  apiKey: "your-ferro-api-key",
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
});

Contributing

We welcome contributions. New providers go in this OSS repo only — never in FerroCloud. See CONTRIBUTING.md for branch strategy, commit conventions, and PR guidelines.

Community

• GitHub Discussions
• Discord
• Built with Ferro Labs AI Gateway? Open a PR to add to our showcase.

License

Apache 2.0 — see LICENSE.