Concurrency modes

Concurrency is a server scheduling capability. It is not tied to one model, one batch width, or one kernel geometry. Each model backend declares which scheduler routes it can install and validates its own shapes and limits when the model loads.

MTPLX keeps model execution on one owner thread. A scheduler decides whether requests run alone or share model work. Even when requests share a batched allocation or forward pass, they keep separate prompt history, sampler state, random-number state, stop state, and logical KV ownership.

Scheduler modes

Value	Behavior
serial	Run one request at a time on the backend's normal generation route; this is the default
cooperative	Interleave independently owned request work where the backend supports it
ar_batch	Batch target-only autoregressive decode on a compatible backend
mtp_batch	Batch native-MTP decode using a model-specific installed MTP lane
mtp_cohort_experimental	Opt into experimental native-MTP cohort scheduling
hyper	Serve one request at a time (extra requests queue FIFO like serial) and reserve batch width for that request's own speculative rows

The mode name is generic. It does not define a batch width, speculative depth, context limit, tensor shape, or numerical policy. Those are properties of the installed model/backend implementation.

hyper

hyper is not a concurrency mode: external admission is fixed at 1, and simultaneous requests queue FIFO behind the active one with exactly the semantics serial gives. What distinguishes it is where width goes — when the width machinery is installed, batch rows hold self-generated speculative branches of the one admitted request, never a second client. At width 1 (the current singleton chassis) a request rides the untouched serial generation path: solo MTP oracle, paged KV cache, custom kernels, compiled verify bank, no padded batch object, no gather window. Because admission is definitional, the server refuses --max-active-requests, --decode-batch-max, a positive --batch-wait-ms, and the agent/throughput presets in this mode at startup. The health payload reports the chassis under scheduler.hyper (admission cap, queue depth, width, and width receipts).

Ownership contract

Every admitted request owns its own:

• prompt and generated tokens;
• logical KV and recurrent state;
• target and draft sampler settings;
• seeded random-number stream;
• token budget, stop state, cancellation event, and output stream.

A backend may place those values in shared batched buffers. Row-specific masks, offsets, commits, and rewinds must still prevent one request from reading or changing another request's context. Fixed-shape lanes may run in lockstep; that is shared scheduling, not shared context.

An optimized lane is installed only after its backend validates the model, geometry, dtype, cache layout, kernels, and construction self-checks. An unsupported combination fails clearly instead of silently changing to AR or a different kernel.

Select a mode

Choose the mode when the server starts. Backend-specific modes may require additional flags:

mtplx serve \
  --model <model-or-path> \
  --scheduler-mode <serial|cooperative|ar_batch|mtp_batch|mtp_cohort_experimental|hyper>

Or save the scheduler choice:

mtplx config set scheduler_mode mtp_batch
mtplx config show --json

Scheduler and kernel routes are construction-time settings. Stop and restart the server after changing them. A CLI value overrides the saved value for that launch. Other required flags depend on the selected backend; use its linked implementation guide instead of copying geometry from another model.

Backend implementations

Model-specific guides record the exact supported features, required launch contract, batch geometry, numerical choices, limitations, and benchmark receipts for each installed lane:

• Qwen3.6 35B A3B fixed B8/K1 MTP lane

This list describes available implementations. It does not redefine the generic concurrency modes or imply that future MTP lanes must use the same width, depth, context limit, or kernels.

Confirm concurrent behavior

The health payload reports the selected scheduler and observed execution:

curl -s http://127.0.0.1:8000/health | jq '.scheduler | {
  mode,
  active_lane,
  config,
  telemetry
}'

The configured mode proves only what was selected. To prove concurrent work actually ran, use the backend guide's behavioral receipt, such as an observed multi-row width or batch histogram after sending simultaneous requests.