modelprobe

June 24, 2026 · View on GitHub

A tiny, dependency-free availability prober for any OpenAI-compatible LLM gateway.

Point it at a base URL + API key and it tells you, per model, is it up and how fast — by making one real (but deliberately tiny) call per model. 2xx = up. Run it after every deploy, on a cron, or from a $5 VM. Single static binary, stdlib only, no config files required.

$ modelprobe -base-url https://api.example.com -api-key sk-xxx
modelprobe https://api.example.com — 🚨 1/4 FAILED (4 models, 6.2s)
  ❌ claude-3-7-sonnet                    HTTP 503: upstream unavailable
  ✅ gpt-4o                               412ms
  ✅ gemini-2.5-pro                       880ms
  ✅ o3-pro                              1240ms (via responses)
$ echo $?
1   # non-zero on any outage → drop it in CI and the pipeline goes red

Why it's built this way (the four principles)

Reuse real traffic — the cheapest call is the one you don't make. If you already know a model is healthy (it served real requests in the last hour), don't pay for a synthetic probe. Wire a SkipModel hook (library) and those models are reported healthy without a call:
```
prober.Config{ SkipModel: func(m string) (bool, string) {
    if liveTrafficHealthy(m) { return true, "live traffic OK (1h)" }
    return false, ""
}}
```
Keep probes short. Default prompt is "Reply with exactly: pong" with max_tokens=16 — cents per full run, not dollars. Only lengthen it (-content, -max-tokens) when you are deliberately testing long-context / long-storage behavior.
Probe from the outside, like a real customer. It talks to your public base URL with a normal bearer key over /v1/chat/completions — no internal IDs, no private bypass. What it sees is what your users see. For true generality it also works against OpenAI, OpenRouter, or any compatible endpoint.
Standalone & shareable. Zero third-party deps, one binary, ~250 lines of core. Fork it, deploy it to a Lambda/VM/CI, share it.

Install / run

# Prebuilt binary (no Go toolchain needed) — grab your platform from the latest release:
#   https://github.com/cuihuan/modelprobe/releases/latest
# …or build it yourself:
go install github.com/cuihuan/modelprobe@latest      # or: go build -o modelprobe .
modelprobe -base-url https://api.example.com -api-key sk-xxx

Base URL and key can come from MODELPROBE_BASE_URL / MODELPROBE_API_KEY instead of the flags.

Flags

flag	default	meaning
`-base-url`	`$MODELPROBE_BASE_URL`	gateway base URL (required)
`-api-key`	`$MODELPROBE_API_KEY`	bearer token
`-models`	(discover)	comma-separated ids; empty → `GET /v1/models`
`-content`	`Reply with exactly: pong`	probe prompt (keep short)
`-max-tokens`	`16`	output cap (cost control)
`-concurrency`	`4`	in-flight probes
`-interval`	`1.6s`	min gap between launches (rate-limit safety)
`-timeout`	`45s`	per-model timeout
`-no-responses-fallback`	`false`	don't retry chat-only failures via `/v1/responses`
`-json`	`false`	machine-readable output
`-webhook`	—	push a Feishu/Lark result card
`-quiet`	`false`	print only on failure
`-version`	—	print build version and exit

Exit codes: 0 = all up · 1 = one or more models down · 2 = usage/config or probe error.

Notes

Rate-limit safe by design. A 1.6s launch interval keeps a 40-model run under a typical 60 req/min per-key limit — a probe storm that 429s itself is worse than no probe at all.
/v1/responses fallback. Models that only speak the Responses API (OpenAI *-pro/codex) are probed there automatically; harmless for gateways without that endpoint.

Deploy (lightweight) — pick by how much infra you want

Tier	What	When
0 · GitHub Actions	`examples/github-actions-probe.yml` → `.github/workflows/`. Free, zero servers, runs on a cron, goes red + Feishu card on outage.	The default. No infra.
1 · Docker / systemd	6 MB scratch image. `examples/docker-compose.yml` (re-probes every 2 min) or `examples/systemd/` (timer on any $5 VM) for a fixed external vantage point + sub-5-min cadence*.	Probe from a specific IP/region, often.
2 · Serverless	`examples/fly.toml` (Fly scheduled machine), or the static binary on Lambda / Cloud Run Job + scheduler. Scale-to-zero.	Specific cloud region, no always-on cost.

# Docker (image has the binary as ENTRYPOINT)
docker build -t modelprobe .
docker run --rm -e MODELPROBE_API_KEY=sk-xxx modelprobe -base-url https://api.example.com

# Or always-on, every 2 min:
MODELPROBE_API_KEY=sk-xxx BASE_URL=https://api.example.com docker compose -f examples/docker-compose.yml up -d

Library use

import "github.com/cuihuan/modelprobe/prober"

res, err := prober.Probe(ctx, prober.Config{BaseURL: "https://api.example.com", APIKey: key})
for _, r := range res { /* r.Model, r.OK, r.LatencyMs, r.Via, r.Err */ }

modelprobe is the lightweight "is each model up and how fast" layer of a small AI-gateway evaluation toolset:

llm-gateway-bench — when "is it up?" isn't enough: a black-box benchmark of gateway behavior (TTFT/throughput, fake-streaming, model-echo, usage inflation, context truncation), with a live leaderboard.
awesome-ai-gateway — a curated list of AI gateways with a reproducible cost benchmark and a compliance/security scorecard, to help you pick one.

License

MIT (or your project's license once extracted to its own repo).