GPU fit planning

Last modified: 2026-08-19

When you name a model, the fit planner decides which quantization to run on the GPU you have, and it refuses a configuration the card cannot serve before an engine ever starts. The goal is a useful message at config time instead of an out-of-memory crash at 2am.

What it answers

Two questions, in order:

1. Can this card even run the quant? A quant that fits by size can still be a kernel the hardware lacks. A Turing card (a cloud T4) has 16 GB but no FP8 and no Marlin int4 kernels, so an FP8 model that would fit is still unrunnable there. The planner reads CUDA compute capability, not just free VRAM, and skips a quant the card cannot execute.
2. Does it fit VRAM at the context you asked for? The estimate is the weight bytes for the chosen quant plus the KV cache at the planned sequence length, times a headroom factor for the CUDA context, activations, and the engine working set.

It walks the model's quant list in preference order and returns the first quant that both runs and fits. If none run, the error says so and names the capability gap. If they run but none fit, the error says that instead, with the smallest estimate it found.

flowchart TD
    START["Model's quant list,\nin preference order"] --> NEXT{Next candidate quant}
    NEXT --> CAP{"Capability gate:\ndoes the card's compute\ncapability have the kernel?"}
    CAP -->|no| SKIPCAP["Skip: capability refusal\n(names the missing kernel)"]
    SKIPCAP --> NEXT
    CAP -->|yes| SIZE["Estimate VRAM:\nweight bytes + KV cache\nat planned seq_len,\ntimes headroom factor"]
    SIZE --> FIT{Fits free VRAM?}
    FIT -->|no| SKIPFIT["Skip: track as smallest\nestimate seen so far"]
    SKIPFIT --> NEXT
    FIT -->|yes| THROUGHPUT["Estimate decode throughput\nfrom memory bandwidth"]
    THROUGHPUT --> PICK["Pick this quant:\nreport quant + estimated VRAM"]
    NEXT -->|list exhausted, none passed capability| FAILCAP["Refuse: capability refusal,\nnames the kernel gap"]
    NEXT -->|list exhausted, some ran but none fit| FAILSIZE["Refuse: size refusal,\nreports free VRAM and\nsmallest estimate found"]

The VRAM math

Weights are the parameter count times bytes per parameter for the quant. FP16 is 2.0 bytes, FP8 is 1.0, and the GGUF K-quants carry block overhead, so Q4_K_M is about 0.56 bytes per weight rather than the nominal 0.5.

The KV cache is 2 x layers x kv_heads x head_dim x bytes x seq_len. The 2 is key plus value. bytes is 2 for an f16 cache and 1 for an fp8 cache. Two consequences worth internalizing: KV grows linearly with context, so a long context can cost more than the weights, and models with grouped-query attention (a small kv_heads) have a much cheaper KV cache than their parameter count suggests. That is why the A3B-class mixture-of-experts models are the self-hosting sweet spot: total parameters set the VRAM, active parameters set the speed.

KV-cache quantization is a lever here, but how much it buys depends on the engine, because not every engine has a kernel for every mode.

kv_quant	vLLM	SGLang	llama.cpp	mistral.rs
auto	weight-quant default	weight-quant default	weight-quant default	weight-quant default
f16	2 bytes	2 bytes	2 bytes	2 bytes
fp8	1 byte (fp8)	1 byte (fp8_e5m2)	1 byte (q8_0)	weight-quant default
int8	1 byte, served as fp8	1 byte, served as fp8_e5m2	1 byte (q8_0)	weight-quant default
int4	1 byte, served as fp8	1 byte, served as fp8_e5m2	0.5 bytes (q4_0)	weight-quant default

So int4 halves the KV term on llama.cpp and does nothing beyond fp8 on the CUDA engines, which expose only fp8 KV variants; mistral.rs takes no KV dtype flag at all, so a low-precision request there keeps the engine default and books no saving. The planner sizes what the engine will really run, and when it substitutes it logs a warning naming the requested mode, the dtype it serves instead, and the engine, so a request for a mode an engine lacks costs you accuracy in the plan rather than an out-of-memory failure at first token.

Quantizing the cache trades a little quality for capacity either way, so it stays opt-in.

Capability tiers

The planner gates FP8 on the compute capability the card reports.

Card	VRAM	Compute capability	FP8	Typical auto pick
T4 (Turing)	16 GB	7.5	no	int4 / GGUF, <=14B
L4 (Ada)	24 GB	8.9	yes	FP8 for a 30B-A3B at short context
A10G (Ampere)	24 GB	8.6	no	int4 / AWQ
A100 (Ampere)	40 or 80 GB	8.0	no	int4 or f16
H100 (Hopper)	80 GB	9.0	yes	FP8

The two cheap cloud GPUs, the T4 and the L4, are the first-class lower-end target. On a T4 the planner refuses FP8 with a capability message and falls back to an int4 or GGUF quant that the card can run. On an L4 it accepts FP8.

Throughput, not just fit

A quant can fit and still be too slow to be useful. The planner also estimates decode throughput from the card's memory bandwidth, since single-stream decode is memory-bandwidth bound: roughly the achievable bandwidth divided by the bytes read per generated token. That catches "this fits but will crawl" before you wait for a load, and it lets the planner prefer the faster of two quants that both fit.

Why did it pick this, or refuse

Every decision is meant to be legible. When it refuses, the message distinguishes the two failure modes: a capability refusal names the kernel the card lacks (FP8 on a Turing T4), and a size refusal reports the free VRAM and the smallest estimate it could find. When it picks, it reports the quant and the estimated VRAM at your context. sbproxy doctor <config> shows the same resolution per model before anything spawns, and sbproxy models prints a per-GPU fit verdict for every catalog model.

A worked example

The built-in catalog lists qwen3-14b with its quants in preference order, [FP8, Q4_K_M]. Name it in a serve: block (the shape is the same as examples/ai-local-serving):

origins:
  "ai.local":
    action:
      type: ai_proxy
      providers:
        - name: local
          default_model: qwen3-14b
          models: [qwen3-14b]
          serve:
            eviction: lru
            cache_budget_gib: 200
            models:
              - model: qwen3-14b   # catalog id; the quant list comes from the catalog
                engine: vllm
                keep_alive: 30m

On an L4 the planner takes the first quant that both runs and fits. FP8 passes the capability gate (compute 8.9), and 14B parameters at 1.0 bytes per weight is about 13 GiB of weights, plus the KV cache at the planned context and the 1.15x headroom factor, inside the card's 24 GB. The plan records the chosen quant, the estimated VRAM, and the context length the estimate assumed.

On a T4 the FP8 candidate is skipped at the capability gate and the walk continues to Q4_K_M, roughly 7.3 GiB of weights, which runs and fits in 16 GB. You only see a refusal when every candidate fails, and the message says which gate failed:

no candidate quant runs on Tesla T4: FP8 needs FP8 kernels but Tesla T4 (compute 7.5) has none
no candidate quant fits 15.0 GiB free on Tesla T4: smallest estimate was 18.2 GiB

The first is a capability refusal: nothing in the quant list can execute on this card. The second is a size refusal: everything could execute, nothing fits, and the planner reports the smallest estimate it found so you know how far off you are.

Related

• model-host.md - the subsystem this is part of.
• self-hosting.md - the getting-started guide.
• model-host-certification.md - the T4/L4 certification the tiers above are checked against.