AI Gateway Examples

Examples demonstrating how to use Ferro Labs AI Gateway — an open-source AI gateway written in Go with a unified API for OpenAI, Anthropic, Bedrock, Azure, and 100+ LLMs.

Directory Structure

.
├── basic/                    # Direct provider request — the simplest use case
├── streaming/                # Streaming chat completions (real-time token output)
├── embeddings/               # Generate embeddings (vectors) for search / RAG
├── config-file/              # Load configuration from a YAML file
├── conditional-routing/      # Route models to providers by request field
├── caching/                  # Cache identical responses (cost / latency savings)
├── custom-plugin/            # Write and register a custom plugin
├── embedded/                 # Embed the gateway inside an existing HTTP server
├── fallback/                 # Automatic fallback between providers on failure
├── loadbalance/              # Weighted load balancing across multiple providers
├── with-circuit-breaker/     # Per-provider circuit breaker configuration
├── with-guardrails/          # Built-in word-filter and token-limit guardrails
├── with-hooks/               # Event hooks for telemetry and audit logging
├── with-mcp/                 # MCP (Model Context Protocol) tool-call integration
└── shared/                   # Shared provider helpers (used by all examples)

Quick Start

Prerequisites

• Go 1.25+ — install
• At least one LLM provider API key (OpenAI, Anthropic, Groq, Gemini, Mistral, Together, Cohere, or DeepSeek)

Run an Example

# Set a provider API key
export OPENAI_API_KEY=sk-...
# or
export ANTHROPIC_API_KEY=sk-ant-...
# or any other supported provider key

# Run the basic example
go run ./basic

Examples

basic

Sends a single chat completion request to the first provider for which an API key is configured. Great starting point.

OPENAI_API_KEY=sk-... go run ./basic

streaming

Streams chat completion tokens in real-time via gw.RouteStream(). Tokens are printed as they arrive instead of waiting for the full response.

OPENAI_API_KEY=sk-... go run ./streaming

embeddings

Generates text embeddings via gw.Embed and prints the cosine similarity between sentences — the building block for semantic search and RAG. Uses OpenAI's text-embedding-3-small.

OPENAI_API_KEY=sk-... go run ./embeddings

config-file

Loads the full gateway configuration from a YAML file with config.LoadConfig instead of building it in Go, and routes using a model alias defined in that file. The production configuration pattern.

OPENAI_API_KEY=sk-... go run ./config-file

conditional-routing

Uses the conditional strategy to route different models to different providers through one endpoint — the basis for cost/tier routing. Requires two provider keys.

OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... go run ./conditional-routing

caching

Enables the built-in response-cache plugin so an identical request is served from memory instead of the provider. Prints per-call latency to show the cache hit.

OPENAI_API_KEY=sk-... go run ./caching

custom-plugin

Demonstrates writing a plugin.Plugin implementation (a request-ID injector) and registering it at the before_request stage.

OPENAI_API_KEY=sk-... go run ./custom-plugin

embedded

Shows how to embed the gateway as an HTTP handler inside your own net/http server, mounting the AI route alongside your existing application routes.

OPENAI_API_KEY=sk-... go run ./embedded
# Then: curl -X POST http://localhost:8080/ai/v1/chat/completions -d '...'

fallback

Configures ModeFallback: the gateway tries providers in order, retrying on the primary before switching to the next. Requires at least one provider key; works better with two.

OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... go run ./fallback

loadbalance

Configures ModeLoadBalance with weighted targets. The first provider receives 70% of requests, the rest share the remaining 30%. Requires at least two provider keys.

OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... go run ./loadbalance

with-circuit-breaker

Attaches a CircuitBreakerConfig to the primary provider target. After two consecutive failures the circuit opens for 10 s, routing all traffic to the fallback.

OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... go run ./with-circuit-breaker

with-guardrails

Loads the built-in word-filter and max-token guardrail plugins via PluginConfig. Requests containing blocked words are rejected before reaching the provider.

OPENAI_API_KEY=sk-... go run ./with-guardrails

with-hooks

Registers an event hook via gw.AddHook(fn). The hook receives gateway.request.completed and gateway.request.failed events asynchronously after each request — useful for telemetry, cost tracking, or audit logging.

OPENAI_API_KEY=sk-... go run ./with-hooks

with-mcp

Starts a local mock MCP server, wires it into the gateway, and sends a tool-calling request. The gateway injects tool definitions, drives the agentic tool-call loop, and returns the final answer.

OPENAI_API_KEY=sk-... go run ./with-mcp
# Works best with models that support tool/function calling (gpt-4o, claude-3-5-sonnet, etc.)

Supported Provider Environment Variables

Variable	Provider
OPENAI_API_KEY	OpenAI
ANTHROPIC_API_KEY	Anthropic
GROQ_API_KEY	Groq
GEMINI_API_KEY	Google Gemini
MISTRAL_API_KEY	Mistral
TOGETHER_API_KEY	Together AI
COHERE_API_KEY	Cohere
DEEPSEEK_API_KEY	DeepSeek

More Information

• AI Gateway repository
• Documentation