Configure SGLang for FrontierAgent

The files in this directory are starting points for running an OpenAI-compatible SGLang server. They are not universal hardware presets: model architecture, checkpoint format, SGLang version, driver, available VRAM, context length, and concurrency all affect whether a configuration fits.

Use .env.sglang.example at the repository root for the small infrastructure smoke test. Use a file in this directory only after the smoke test passes:

Template	Intended starting point	Important caveat
35b-4090.env.example	One 24 GB RTX 4090	Needs a 4-bit export; measure its limits
35b-5090.env.example	One 32 GB RTX 5090	Values certified against a GPTQ-Int4 MoE export, which is also the only format that fits this card
35b-multigpu.env.example	Two GPUs with tensor parallelism	Needs a 4-bit export; validate peer access and per-card VRAM

All three name apodex/Apodex-1.1-mini in SGLANG_MODEL_ID, and that repository publishes FP16 weights — roughly 70 GB for a 35B checkpoint, before any KV cache. None of these consumer-GPU templates can load that as-is: quantize first and point SGLANG_LOCAL_MODEL_PATH at the export (it takes precedence over the repository id), or substitute a 4-bit repository. The model id is there to say which model to serve, not to promise it fits.

Quick start

For Docker on a Linux NVIDIA host:

cp config/sglang/35b-5090.env.example .env.sglang
chmod 600 .env.sglang
$EDITOR .env.sglang

./docker/run-sglang.sh doctor
./docker/run-sglang.sh up
./docker/run-sglang.sh smoke
./docker/run-sglang.sh tui

For a provider-owned GPU container without nested Docker, use the same env file with scripts/run-sglang-native.py; follow the native SGLang guide.

Do not commit .env.sglang. It may contain HF_TOKEN and web-tool credentials. The example files intentionally contain no secrets.

How the env file is used

The env file configures two different layers:

1. docker/sglang_entrypoint.py translates the model settings into python -m sglang.launch_server arguments.
2. Compose or the native launcher translates the client token limits into OPENAI_* variables consumed by the FrontierAgent workflow profile.

That distinction matters: SGLANG_MAX_INPUT_TOKENS and SGLANG_MAX_OUTPUT_TOKENS limit FrontierAgent requests; they are not SGLang server flags. SGLANG_CONTEXT_LENGTH, by contrast, becomes SGLang's --context-length and is also advertised to FrontierAgent.

Variable reference

Model and API identity

Variable	Meaning
SGLANG_PROFILE	Human-readable label for recording the chosen configuration; the launcher does not currently consume it or select settings from it
SGLANG_IMAGE	Docker image containing the pinned SGLang runtime; Docker path only
SGLANG_MODEL_ID	Hugging Face repository ID passed as --model-path
SGLANG_LOCAL_MODEL_PATH	Existing checkpoint directory; when set, it takes precedence over SGLANG_MODEL_ID and is mounted read-only in Docker
SGLANG_SERVED_MODEL_NAME	Stable name returned by /v1/models and sent by FrontierAgent, independent of the checkpoint path
HF_TOKEN	Token for gated/private Hugging Face repositories; leave empty for public models

A local checkpoint must contain the model config, tokenizer, index, and every referenced weight shard. Review a model repository before setting SGLANG_TRUST_REMOTE_CODE=1.

GPU, context, and request budgets

Variable	Meaning
SGLANG_TP_SIZE	SGLang tensor-parallel size (--tp-size); normally the number of GPUs sharding one model
SGLANG_GPU_COUNT	Number of GPUs reserved by Docker Compose; it is not passed to SGLang
SGLANG_CONTEXT_LENGTH	Maximum server context (--context-length), including input and generated tokens
SGLANG_MAX_INPUT_TOKENS	FrontierAgent input guard, exported as OPENAI_MAX_INPUT_TOKENS
SGLANG_MAX_OUTPUT_TOKENS	Maximum generation per agent call, exported as OPENAI_MAX_TOKENS
SGLANG_MEM_FRACTION_STATIC	Fraction of GPU memory reserved by SGLang for weights and the KV-cache pool
SGLANG_EXTRA_ARGS	Additional sglang.launch_server arguments, parsed as a shell-style argument string

Keep these invariants true:

SGLANG_MAX_INPUT_TOKENS + SGLANG_MAX_OUTPUT_TOKENS
    <= SGLANG_CONTEXT_LENGTH

SGLANG_MAX_INPUT_TOKENS
    > 0.8 * SGLANG_CONTEXT_LENGTH

The second invariant lets FrontierAgent's tiered compaction trigger at 80% of the workflow context before its hard input guard aborts the request. The doctor checks both invariants.

Longer context and more simultaneous requests require more KV-cache memory. On the first successful boot, keep --max-running-requests 1; raise context or concurrency one at a time and record peak VRAM and latency. If startup or long prefill runs out of memory, first lower SGLANG_MEM_FRACTION_STATIC slightly, reduce SGLANG_CONTEXT_LENGTH, or add a smaller --chunked-prefill-size through SGLANG_EXTRA_ARGS. A lower static fraction leaves more runtime headroom but also shrinks the KV-cache pool.

For multi-GPU runs, SGLANG_TP_SIZE must not exceed SGLANG_GPU_COUNT. Tensor parallelism also depends on GPU peer access and interconnect performance; two cards do not automatically provide twice the usable context or throughput.

Model-specific parsing and precision

Variable	Meaning
SGLANG_TOOL_CALL_PARSER	Parser matching the model's emitted tool-call format (--tool-call-parser)
SGLANG_REASONING_PARSER	Parser matching the model's reasoning format (--reasoning-parser)
SGLANG_CHAT_TEMPLATE	Override the checkpoint's packaged chat template (--chat-template). Leave empty to use the packaged one. In Docker the path must resolve inside the container, so point it at a file under the mounted model directory — a host path such as /mnt/…/templates/x.jinja is not visible there
SGLANG_DTYPE	Weight/activation dtype (--dtype); auto follows checkpoint metadata
SGLANG_QUANTIZATION	Explicit SGLang quantization backend (--quantization) when the checkpoint requires one
SGLANG_TRUST_REMOTE_CODE	Adds --trust-remote-code when set to 1, true, yes, or on

Parser names are model-family specific. A server can be healthy and return text while structured tool calls still fail. Use ./docker/run-sglang.sh smoke after changing the model, chat template, tool parser, or reasoning parser.

The native doctor also enforces a per-model minimum SGLang version from compatibility.json, matched by prefix against SGLANG_MODEL_ID. Apodex checkpoints are Qwen3.5-architecture and inherit the same floor, but apodex/… does not match Qwen/Qwen3.5, so the vendor path carries its own entry. Serving these weights from a differently-named repository means adding a prefix there too, or the version floor silently does not apply.

Prefer checkpoints that are already quantized. Do not infer the backend only from a filename: read the checkpoint's model card and compare it with the SGLang version's supported quantization methods. Leaving SGLANG_QUANTIZATION empty lets SGLang inspect checkpoint metadata; set it only when the format or model guidance requires an explicit backend.

Docker and native launcher controls

Variable	Meaning
SGLANG_PORT	Loopback host port in Docker and listening port in native mode; default 30000
SGLANG_RESTART_POLICY	Docker Compose restart policy; no avoids unexpected GPU use after reboot
APODEX_DOCKER_SUBNET	Optional explicit Compose subnet for VPN/corporate-network conflicts
SGLANG_BUILD_AGENT	Build FrontierAgent from this checkout instead of pulling its release image; does not rebuild SGLang
SGLANG_PYTHON	Native-mode Python executable in the environment where import sglang succeeds
SGLANG_EXPECTED_VERSION	Optional native doctor assertion; it does not install or change SGLang
SGLANG_DOWNLOAD_DIR	Native SGLang/Hugging Face download directory, ideally on persistent storage
SGLANG_NATIVE_STATE_DIR	Native launcher's PID/log state directory
SGLANG_NATIVE_HOST	Native bind address; keep 127.0.0.1 unless an unauthenticated endpoint must be exposed deliberately
HF_HOME, HF_HUB_CACHE, HF_HUB_OFFLINE	Hugging Face cache placement and offline behavior for native mode

SERPER_* and JINA_* configure FrontierAgent web tools, not SGLang. Local inference and the TUI do not require those keys, but research tools may.

A safe tuning sequence

1. Run the root smoke profile to verify the driver, Docker GPU passthrough, SGLang API, tool parsing, and FrontierAgent connection.
2. Select the exact checkpoint and confirm that its architecture, quantization, and parser names are supported by the pinned SGLANG_IMAGE version.
3. Start at one running request and a conservative context window.
4. Run doctor, then up, inspect logs, and run smoke.
5. Exercise representative long prompts while watching nvidia-smi.
6. Increase context, then concurrency, one variable at a time. Record the image digest, checkpoint revision, GPU model/count, peak VRAM, and verified limits.

Do not describe a candidate template as supported hardware until that exact combination has passed representative workloads on the physical GPU.

Upstream SGLang references

• Server arguments
• Quantization
• Tool parser
• Reasoning parser
• Supported models

The installed version is authoritative. Run the following with the same Python environment or container image used to serve the model:

python -m sglang.launch_server --help

For end-to-end setup and troubleshooting, see Linux NVIDIA + Docker, Linux NVIDIA native, and the GPU compatibility matrix.