autobatcher (TypeScript)

Drop-in OpenAI client that transparently batches requests via the Batch API. Designed for the Doubleword Inference API where batch pricing saves up to 90%.

BatchOpenAI is a subclass of OpenAI — it passes instanceof checks and works anywhere the standard client is accepted. The only difference: chat.completions.create() and embeddings.create() calls are collected into a queue and submitted as batch jobs instead of making individual HTTP requests.

Installation

npm install autobatcher openai

Usage

Chat completions

import { BatchOpenAI } from "autobatcher";

const client = new BatchOpenAI({
  apiKey: "sk-...", // or set OPENAI_API_KEY env var
  baseURL: "https://api.doubleword.ai/v1",
});

const response = await client.chat.completions.create({
  model: "Qwen/Qwen3.5-35B-A3B-FP8",
  messages: [{ role: "user", content: "What is 2+2?" }],
});
console.log(response.choices[0].message.content);

await client.close();

Embeddings

const response = await client.embeddings.create({
  model: "Qwen/Qwen3-Embedding-8B",
  input: "Hello, world!",
});
console.log(response.data[0].embedding.slice(0, 5));

Parallel requests

The real power comes when you have many requests:

const prompts = ["What is 1+1?", "What is 2+2?", "What is 3+3?"];

// All requests are batched together automatically
const results = await Promise.all(
  prompts.map((prompt) =>
    client.chat.completions.create({
      model: "Qwen/Qwen3.5-35B-A3B-FP8",
      messages: [{ role: "user", content: prompt }],
    })
  )
);

for (const r of results) {
  console.log(r.choices[0].message.content);
}

await client.close();

Serve mode

autobatcher serve runs a local OpenAI-compatible HTTP proxy that batches incoming requests. Useful for transparently batching traffic from tools that support a custom baseURL — evaluation frameworks, benchmark runners, or any OpenAI SDK consumer.

npx autobatcher serve \
  --base-url https://api.doubleword.ai/v1 \
  --api-key "$DOUBLEWORD_API_KEY" \
  --port 8080 \
  --batch-size 1024 \
  --batch-window 60 \
  --completion-window 24h

Then point any OpenAI-compatible client at the proxy:

export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=dummy

Use your real credential for the proxy's upstream --api-key. The downstream client uses a dummy key because it is only talking to the local proxy.

Supported proxy routes:

Route	Upstream batched endpoint
POST /v1/chat/completions	/v1/chat/completions
POST /v1/embeddings	/v1/embeddings
POST /v1/responses	/v1/responses
GET /health	local healthcheck

The proxy emits structured JSON lifecycle events to stdout for log collection:

{"source":"autobatcher","event":"server_started","ts":1776163751.821,"host":"127.0.0.1","port":8080}

Programmatic usage

You can also start the server programmatically:

import { serve } from "autobatcher";

const { server, close } = serve({
  baseURL: "https://api.doubleword.ai/v1",
  apiKey: "sk-...",
  port: 8080,
  batchSize: 1024,
  completionWindow: "1h",
});

// Later: gracefully shut down
await close();

Configuration

Parameter	Default	Description
apiKey	env var	OpenAI / Doubleword API key (falls back to OPENAI_API_KEY)
baseURL	provider default	API base URL
batchSize	1000	Submit batch when this many requests are queued
batchWindowSeconds	10	Submit batch after this many seconds
pollIntervalSeconds	5	How often to poll for batch completion
completionWindow	"1h"	Completion deadline (see below)

Completion window

The completionWindow controls the deadline and pricing tier:

• "1h" (default) — async inference. Faster turnaround than batch mode, still significantly cheaper than real-time. Supported by the Doubleword Inference API only.
• "24h" — batch inference. Maximum cost savings (up to 90% with the Doubleword Inference API, 50% with OpenAI). Use for background jobs like evals, data processing, or bulk extraction where latency doesn't matter. This is the only window OpenAI supports.

Supported endpoints

Endpoint	Return type
client.chat.completions.create()	ChatCompletion
client.embeddings.create()	CreateEmbeddingResponse

All other methods on the client (e.g. client.models.list(), client.files.create()) pass through to the underlying OpenAI client unchanged — only the endpoints above are intercepted for batching.

Limitations

• Not suitable for real-time or interactive use cases — batch mode adds latency from the collection window and polling cycle.
• Streaming is not supported. Requests with stream: true will have streaming stripped and results returned as a complete response.
• OpenAI only supports completionWindow: "24h". The "1h" window is a Doubleword-specific feature.
• No automatic escalation to real-time if the completion window elapses — the batch will be marked as expired.
• Responses API batching (client.responses.create()) is available via the serve proxy but not yet via the BatchOpenAI class directly.

License

MIT