AI Backends

September 19, 2026 · View on GitHub

AIBackends is an API server that you can use to integrate AI into your applications. You can run it locally or self-host it.

The project supports running open models locally with Ollama, LM Studio or LlamaCpp. It also supports LLM Gateway, OpenRouter, OpenAI, Anthropic, Google AI Studio, Baseten and ZAI providers.

Beyond ready-to-use endpoints for text, vision, and data tasks, AIBackends now ships AI Agents: autonomous, tool-using agents you can run as one-off tasks or chat with in multi-turn sessions that remember the whole conversation. See Agents below.

Why AI Backends?

The purpose of this project is to make common AI use cases easily accessible to non-coders who want to add AI features to their applications. AIBackends have been tested with popular AI app builder tools like Bolt.new, v0 and Lovable. You can also use it with Warp, Cursor, Claude Code, Windsurf or AmpCode.

Since APIs are ready to use, you don't need to understand prompt engineering. Just prompt the API documentation and you are good to go. For those who want use with online app builders, you need to host AIBackends on your own server. I have tested in Railway and it is a good option.

AI Backends

Agents (New)

AIBackends can now run autonomous, tool-using agents. Instead of a single prompt/response, an agent plans across multiple turns: it decides which tools to call, reads the results, and keeps going until the task is done.

Multi-turn chat with sessions — POST /api/agent/chat holds a conversation. The agent remembers everything said so far, so follow-ups like "book the first one" or "did my last payment go through?" just work. Sessions are kept in memory with a 30-minute idle expiry.
One-off tasks — POST /api/agent/run completes a task and returns the final answer plus every tool call the agent made.
Scenarios — pick a toolset per request: general (calculator, date/time, weather), customer-support (account, subscription, billing, and ticket tools), or real-estate (search listings, property details, and viewing appointment booking). Ships with demo data so you can try it immediately.
Live streaming — both endpoints support SSE so you can watch turns, tool calls, and the reply stream in real time.

Start a chat, then continue it with the returned sessionId:

curl --location 'http://localhost:3000/api/v1/agent/chat' \
--header 'Content-Type: application/json' \
--data '{
    "payload": {
        "message": "Hi, I am jane.cruz@example.com. Is my subscription active?",
        "scenario": "customer-support"
    },
    "config": {
        "provider": "openrouter",
        "model": "deepseek/deepseek-v4-flash"
    }
}'

# Response includes "sessionId" — send it with the next message to continue the conversation:
# { "payload": { "message": "Did my last payment go through?", "sessionId": "<sessionId>" }, ... }

Requires OpenRouter (OPENROUTER_API_KEY) or OpenAI (OPENAI_API_KEY) with a tool-calling model.

You can also create your own agents and tools — via the API or the admin dashboard — without touching code. Define a custom HTTP tool (one request per call, with a JSON Schema for its arguments), compose agents from built-in and custom tools, and they become immediately runnable as scenarios and selectable in the demos. See Admin Dashboard.

Full usage guide and API shapes: docs/agents-api.md
Custom agents, tools, and keys: docs/admin-api.md
Interactive demos: Agent Chat (multi-turn) and Agent Tasks (one-off)
Endpoint reference: Agents section under Available APIs

Admin Dashboard

The main admin dashboard — separate from the demos — lives at http://localhost:3000/admin. Use it to:

Create and edit agents: system prompt, toolset, skills, and sample tasks; new agents are instantly runnable via the Agents API and listed in the demo pages.
Define custom HTTP tools: point a tool at any HTTP endpoint (with {placeholder} URL templating, query/body argument mapping, and custom headers), then test it right in the dashboard before agents use it.
Create skills: instruction packages (Agent Skills style) whose descriptions are always visible to agents while full content loads on demand via the use_skill tool.
Connect MCP servers: register Model Context Protocol servers (Streamable HTTP or SSE); their tools are auto-discovered as mcp_<server>_<tool> and can be attached to agents like any other tool. A demo MCP server ships in examples/mcp-demo-server.ts.
Configure provider API keys at runtime: set or replace keys for OpenAI, Anthropic, OpenRouter, Google, and more without restarting the server. Keys set here override environment variables and are masked in all responses.

Configuration persists to data/admin-config.json (gitignored — it contains API keys, so protect it like a .env file). All admin APIs require the bearer token in production; enter it once in the dashboard header. Details in docs/admin-api.md.

Supported LLM Providers

Local Providers

Provider	Description	Status
Ollama	Local models (self-hosted)	Available
LM Studio	Local models via OpenAI-compatible API (self-hosted)	Available
LlamaCpp	Local models via llama.cpp server (self-hosted)	Available

Cloud Providers

Provider	Description	Status
LLM Gateway	Recommended - Unified API for multiple LLM providers with free models	Available
OpenAI	GPT models	Available
Anthropic	Claude models	Available
OpenRouter	Open source and private models	Available
Vercel AI Gateway	Open source and private models	Available
Google AI Studio	Gemini models via OpenAI-compatible interface	Available
Baseten	Cloud-hosted ML models with OpenAI-compatible API	Available
ZAI	GLM models with vision/OCR capabilities	Available

Evaluation / Decision Providers

These are not text-generation models. They answer typed questions about a piece of state with calibrated probabilities and are only available through the Evaluation API.

Provider	Description	Status
TypeSafe Jev	System One decision model: `choice`, `score`, and `noul` questions	Available

Set up environment variables

Create a .env file in the root directory of this project and configure your preferred AI services:

# General Configuration
DEFAULT_ACCESS_TOKEN=your-secret-api-key

# CORS Configuration
CORS_ALLOWED_ORIGINS=http://localhost:3000,https://example.com,https://*.example.com

# OpenAI Configuration
OPENAI_API_KEY=your-openai-api-key

# Anthropic Configuration
ANTHROPIC_API_KEY=your-anthropic-api-key

# Google Gemini Configuration
GOOGLE_AI_API_KEY=your-google-ai-api-key
GEMINI_MODEL=gemini-2.5-flash-lite

# Ollama Configuration
OLLAMA_ENABLED=true
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_TIMEOUT=30000

# You can change OLLAMA_BASE_URL to use a remote Ollama instance
 
# LlamaCpp Configuration
LLAMACPP_BASE_URL=http://localhost:8080

# You can change LLAMACPP_BASE_URL to use a remote LlamaCpp instance
 
# LM Studio Configuration 
LMSTUDIO_ENABLED=true
LMSTUDIO_BASE_URL=http://localhost:1234

# You can change LMSTUDIO_BASE_URL to use a remote LM Studio instance

# OpenRouter Configuration 
OPENROUTER_API_KEY=your-openrouter-api-key

# Baseten Configuration
BASETEN_API_KEY=your-baseten-api-key
BASETEN_BASE_URL=https://inference.baseten.co/v1

# LLM Gateway Configuration (Recommended)
LLM_GATEWAY_API_KEY=your-llm-gateway-api-key

# ZAI Configuration (for Vision/OCR endpoints)
ZAI_API_KEY=your-zai-api-key

# TypeSafe / Jev Configuration (for the /api/evaluate decision endpoint)
TYPESAFE_API_KEY=your-typesafe-api-key
TYPESAFE_BASE_URL=https://api.typesafe.ai
TYPESAFE_MODEL=jev-latest
TYPESAFE_TIMEOUT=10000

LLM Gateway Setup (Recommended for Cloud Providers)

LLM Gateway provides a unified API to access multiple LLM providers with a single API key. It includes several free models to get started.

Sign up at LLM Gateway
Get your API key from the dashboard
Set LLM_GATEWAY_API_KEY in your .env file

Important: Make sure to add .env to your .gitignore file to avoid committing sensitive information to version control.

Run the project

You can configure API keys for different AI providers in the .env file.

# Install dependencies
bun install

# Run in development mode and bypasses access token check in the API, do run using this command in production. Always use production when deploying so access token is required. NODE_ENV=development is set in package.json when you run in development mode.
bun run dev

# Build for production
bun run build

Run with Docker

Using a single container (recommended)

Right now this only works with OpenAI, Anthropic and OpenRouter since the docker container

Build the image:

docker build -t ai-backends .

Run the container in the background (loads variables from your .env):

docker run --env-file .env -p 3000:3000 ai-backends &

Set this in your .env file if you're using for development with Ollama in your local machine.

NODE_ENV=development
OLLAMA_BASE_URL=http://host.docker.internal:11434

If deploying to production, set this in your .env file:

NODE_ENV=production
DEFAULT_ACCESS_TOKEN=your-secret-api-key
OPENAI_API_KEY=your-openai-api-key
ANTHROPIC_API_KEY=your-anthropic-api-key
OPENROUTER_API_KEY=your-openrouter-api-key
GOOGLE_AI_API_KEY=your-google-ai-api-key

You need to configure at least one provider api key. Otherwise, the app will not start.

Using Docker Compose

This will run AI Backends API server and Ollama containers using Docker

Ensure you have a .env configured as described in "Set up environment variables" below. You must set DEFAULT_ACCESS_TOKEN and at least one provider credential (or enable a local provider such as Ollama).
Start all services:

docker compose --env-file .env up -d --build

Adding more models to Ollama container

To add more models, you can edit the ollama service command in docker-compose.yml.

For example, to add gemma3:4b, llama3.2:latest and llama3.2-vision:11b models, you can add the following to the ollama service command:

command: -c "ollama serve & sleep 5 && ollama pull gemma3:270m && ollama pull gemma3:4b && ollama pull llama3.2:latest && ollama pull llama3.2-vision:11b && wait"

You might need to adjust the timeout to give enough time for the models to be pulled.

 healthcheck:      
      timeout: 120s //increase this if you're adding more models

Useful commands:

View logs: docker compose logs -f app
Stop/remove: docker compose down

Notes

With Docker Compose, the app container can reach the Ollama service over the compose network (service name: ollama, port: 11434).
You can customize which models are pulled by editing the ollama service command in docker-compose.yml.

aibackends-python

This repo also includes an optional FastAPI service that wraps the local Python library donvito/aibackends (GPU/CPU tasks via llamacpp / transformers).

Code: aibackends-python/
Docs: aibackends-python/README.md
Default URL: http://localhost:8000 (OpenAPI at /docs)

# aibackends-python only
docker compose up aibackends-python --build

# Or locally
cd aibackends-python && pip install -r requirements.txt && uvicorn app.main:app --port 8000

The TypeScript API (port 3000) and aibackends-python (port 8000) are separate services. Use aibackends-python for library tasks such as local summarize/classify/embed/PII/invoice extraction; keep the TypeScript API for the existing multi-provider HTTP endpoints.

Interactive demo: http://localhost:3000/api/v1/aibackends-python-demo
Same-origin proxy: /api/v1/local/* → aibackends-python
Curl / JS examples: examples/aibackends-python.md

Available APIs

Text Processing

Endpoint	Description
/api/summarize	Summarize long text content into concise, key points
/api/translate	Translate text between different languages
/api/sentiment	Analyze the emotional tone and sentiment of text
/api/keywords	Extract important keywords and phrases from text
/api/email-reply	Generate professional email responses based on context
/api/ask-text	Ask questions about provided text and get intelligent answers
/api/highlighter	Identify and highlight the most important information in text
/api/meeting-notes	Transform meeting notes into structured summaries
/api/project-planner	Create detailed project plans with steps, timelines, and considerations
/api/rewrite	Rewrite text with instructions (improve, shorten, fix grammar, tone)
/api/compose	Compose short-form text given a topic
/api/pdf-summarizer	Extract and summarize content from PDF documents with AI
/api/web-search	Perform web searches and get AI-powered summaries of results

Text Processing Examples

Web Search - Search the web and get AI-powered summaries:

Web Search Example

Keywords Extraction - Extract important keywords from text:

Keywords Example

Sentiment Analysis - Analyze emotional tone of text:

Sentiment Example

Translation - Translate text between languages:

Translate Example

Data Generation

Endpoint	Description
/api/synthetic-data	Generate realistic synthetic data based on prompts with optional JSON schema validation

Synthetic Data Generation - Generate realistic test data with custom schemas:

Synthetic Data Example

Image Processing

Endpoint	Description
/api/vision	Analyze images with vision AI - ask questions, detect objects, get coordinates (ZAI GLM-4.6v)
/api/ocr	Extract structured data from images using OCR with optional JSON schema output (ZAI GLM-4.6V)

Vision AI Examples

AIBackends includes powerful vision capabilities powered by ZAI GLM-4.6v models. You can ask questions about images, detect objects, and extract structured data.

Vision Q&A - Ask questions about any image:

Vision Example

OCR Extraction - Extract structured data from documents, receipts, and invoices:

OCR Example

Agents

Autonomous, tool-using agents that plan across multiple turns. Pick a scenario to select the agent's toolset: general (calculator, date/time, weather), customer-support (account, subscription, billing, and ticket tools), or real-estate (search listings, property details, and viewing appointment booking).

Endpoint	Description
/api/agent/run	Run a one-off agent task and get the final answer plus every tool call it made
/api/agent/chat	Chat with an agent in a multi-turn session; the agent remembers the whole conversation
/api/agent/sessions/{id}	Inspect (GET) or end (DELETE) a chat session
/api/agent/scenarios	List available scenarios with their tools and sample tasks
/api/agent/tools	List the general-purpose toolset

Both run and chat support streaming (SSE) so you can watch the agent's turns, tool calls, and reply in real time. Chat sessions are stored in memory with a 30-minute idle expiry. Supported providers: OpenRouter and OpenAI (tool-calling models required).

See the full usage guide and API shapes in docs/agents-api.md, or try the interactive demos: Agent Chat and Agent Tasks.

Evaluation / Decision

Fast, structured decisions for your code instead of generated text. Send a shared state (a string or any JSON object/array) plus a map of typed questions to a System One decision model (TypeSafe Jev) and get back one calibrated answer per question. Keep each question a small, atomic judgment and compose them in code.

Endpoint	Description
/api/evaluate	Answer `choice`, `score`, and `noul` questions about a state with probabilities

Question	Asks for	Answer
`choice`	One option from a set you define	`choice`, `probabilities` per option, `confidence`
`score`	A position on an ordered scale you define	fractional `score`, `legend`, `probabilities`, `confidence`
`noul`	Yes or no	`noul`, the probability (0–1) of yes

curl -X POST http://localhost:3000/api/v1/evaluate \
  -H "Authorization: Bearer $DEFAULT_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "payload": {
      "state": { "user_request": "Create an invoice for Acme Corp" },
      "questions": {
        "route": {
          "type": "choice",
          "instructions": "Which agent should handle this request?",
          "criteria": {
            "accounting": "Invoices and bookkeeping",
            "research": "Research and documents",
            "coder": "Software development",
            "human": "Ambiguous or unsupported"
          }
        },
        "needs_clarification": {
          "type": "noul",
          "instructions": "Is information required before this request can be executed?"
        }
      }
    },
    "config": { "provider": "typesafe", "model": "jev-latest" }
  }'

{
  "provider": "typesafe",
  "model": "jev-latest",
  "answers": {
    "route": {
      "type": "choice",
      "choice": "accounting",
      "probabilities": { "accounting": 0.94, "research": 0.02, "coder": 0.01, "human": 0.03 },
      "confidence": 0.91
    },
    "needs_clarification": { "type": "noul", "noul": 0.18 }
  },
  "usage": { "input_tokens": 123, "output_tokens": 20, "total_tokens": 143 }
}

Try it interactively in the Jev Playground, which ships with presets for agent routing, support ticket triage, prompt injection guarding, PR risk review, and invoice compliance.

Set TYPESAFE_API_KEY in .env or add the key under API Keys in the Admin Dashboard. Evaluation providers are kept separate from the generative LLM providers: typesafe is not accepted by text endpoints such as /api/summarize, and LLM providers are not accepted by /api/evaluate. Upstream 429/529 responses are retried with exponential backoff.

Jev can also be reached through Vercel AI Gateway instead of calling TypeSafe directly: set AI_GATEWAY_API_KEY and use "config": { "provider": "aigateway", "model": "typesafe-ai/jev" }. The request and response shapes are identical — the gateway's boolean answers are translated back to noul, and score legend is reconstructed from the question criteria.

More to come...check swagger docs for updated endpoints.

Tech Stack

Hono for the API server
Typescript
Zod for request and response validation
pi-ai for AI integration
Docker for containerization

Swagger Docs

After running the project, you can access the swagger docs at:

http://localhost:3000/api/ui

Swagger Documentation

Demos

See examples how to use the APIs

You can access demos at http://localhost:3000/api/demos

Demos

Provider and Model Selection

You need to send the service and model name in the request body. See examples in the swagger docs.

For example, to summarize text using Gemini model with Google as provider, you can use the following curl command:

curl --location 'http://localhost:3000/api/v1/summarize' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
    "payload": {
        "text": "Text to summarize",
        "maxLength": 100
    },
    "config": {
        "provider": "google",
        "model": "gemini-2.5-flash-lite",
        "temperature": 0
    }
}'

Or with Ollama:

curl --location 'http://localhost:3000/api/v1/summarize' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
    "payload": {
        "text": "Text to summarize",
        "maxLength": 100
    },
    "config": {
        "provider": "ollama",
        "model": "gemma3:270m",
        "temperature": 0
    }
}'

Available Tools

Home Page: http://localhost:3000/
Swagger Docs: http://localhost:3000/api/ui. You can test the API endpoints here.
JSON Editor: http://localhost:3000/api/jsoneditor
LLM-Friendly API Docs: http://localhost:3000/api/llms.txt. Copy the contents of this file and paste it into AI builder tools like Bolt.new, v0, Lovable, or AI coding assistants to help them understand and use the AIBackends API endpoints.

LLMs.txt Example

Testing Examples

Check swagger docs for examples.

Run the unit tests with bun run test. The TypeSafe integration test is opt-in and only runs when a real key is provided:

TYPESAFE_API_KEY=... bun run test:integration

The project is in active development. More endpoints and providers will be added in the future. If you want to support me with API credits from your provider, please contact me.

I am also open to sponsorship to support the development of the project.

AI Backends

Why AI Backends?

Agents (New)

Admin Dashboard

Supported LLM Providers

Local Providers

Cloud Providers

Evaluation / Decision Providers

Set up environment variables

LLM Gateway Setup (Recommended for Cloud Providers)

Run the project

Run with Docker

Using a single container (recommended)

Using Docker Compose

Adding more models to Ollama container

aibackends-python

Available APIs

Text Processing

Text Processing Examples

Data Generation

Image Processing

Vision AI Examples

Agents

Evaluation / Decision

Tech Stack

Swagger Docs

Demos

Provider and Model Selection

Available Tools

Testing Examples

Vision

Technical Architecture

Star History

Supporting the project

License