Oxide Infer is a GPU operator runtime for Rust inference engines. It defines operator contracts, chooses an explicit provider and algorithm, freezes an immutable plan, checks runtime resources, and retains them until GPU work settles.

cuda-oxide compiles native Rust device code. Vendor providers such as cuBLASLt enter through the same checked command runtime without passing through the native-kernel toolchain.

Oxide Infer is an operator layer. Consumer engines retain model graphs, continuous batching, request scheduling, KV-cache policy, distributed control, tokenizers, and serving APIs.

One execution model

Every operator follows one lifecycle:

Spec
  → Provider
  → Algorithm
  → Plan
  → Operands
  → CommandScope
  → Completion

Planning fixes the provider, algorithm, workspace contract, launch configuration, artifact, and CUDA Graph policy. Enqueue does not tune, switch providers, or select a silent fallback.

Architecture

flowchart TB
  Engines["Consumer engines<br/>Mistral.rs POC · vLLM planned · custom Rust"]
  Adapter["Engine adapter"]

  subgraph Core["oxide-infer · contracts"]
    Families["attention · gemm · kv_cache<br/>normalization · position · activation<br/>sampling · speculation · moe"]
    Spec["Spec · errors · capabilities · CPU reference"]
  end

  subgraph Cuda["oxide-infer-cuda · execution"]
    Planning["Provider → Algorithm → immutable Plan"]
    Runtime["Operands → CommandScope → Completion"]
    Native["Oxide native provider"]
    Vendor["Vendor providers"]
  end

  Oxide["cuda-oxide<br/>Rust → PTX/cubin"]
  Libraries["cuBLASLt · future vendor libraries"]
  Driver["CUDA Driver · NVIDIA GPU"]
  Lab["oxide-infer-lab<br/>correctness · Graph · sanitizer · performance · engine gates"]

  Engines --> Adapter --> Families --> Spec --> Planning --> Runtime
  Runtime --> Native --> Oxide --> Driver
  Runtime --> Vendor --> Libraries --> Driver
  Lab -. qualifies .-> Core
  Lab -. qualifies .-> Cuda

The native provider owns architecture-specific Rust kernels. sm90a, sm100a, and later modules combine the CUDA primitives that each algorithm needs. TMA is a data-movement primitive. WGMMA and tcgen05 are compute instruction families. They are not runtime providers by themselves.

The architecture document defines ownership, planning, workspace, stream, Graph, artifact, and engine-adapter boundaries.

Operator surface

The table separates source presence from qualification. A source path does not prove device correctness or performance.

Family	Current source	State
Attention	Single decode, paged decode, ragged prefill, paged prefill	Implemented; path-specific requalification
KV cache	Paged append and RoPE plus paged append	Implemented; ownership requalification
GEMM	Contiguous BF16 dense through cuBLASLt	Implemented
GEMV	Native BF16 M=1 SM90a algorithm	Experimental
Normalization	RMSNorm for F32, FP16, and BF16	Implemented; path-specific requalification
Position	BF16 NeoX RoPE with explicit positions	Implemented; path-specific requalification
Activation	SwiGLU and fused epilogues	Planned
Sampling	Logits transforms, RNG, and token selection	Planned
Advanced attention	MLA and expanded KV layouts	Planned
Matrix operations	FP8, grouped GEMM, and MoE shapes	Planned

The operator catalog records the exact dtype, shape, layout, provider, algorithm, and evidence state for every admitted path.

Providers

Oxide Infer exposes provider selection before execution.

Oxide
  native Rust kernels
  → cuda-oxide
  → PTX or cubin
  → CUDA Driver

CublasLt
  checked vendor plan
  → cuBLASLt
  → CUDA Driver

The command runtime gives both paths the same resource and failure model:

• caller-selected CUDA context and stream
• typed read and write regions
• checked span, alignment, alias, and capacity rules
• explicit workspace requirements
• completion-owned resource leases
• typed device-status failures
• fixed-address CUDA Graph capture where admitted

Workspace

Crate	Responsibility
oxide-infer	Backend-independent contracts, errors, capabilities, and CPU references
oxide-infer-cuda	Planning, CUDA command runtime, native kernels, Graphs, and vendor providers
oxide-infer-lab	Non-published H20 gates, matched benchmarks, fixtures, and evidence generation

The workspace does not split GEMM, kernels, or runtime into additional crates. They remain modules until they need a separate dependency, release, ownership, or safety boundary.

Build and validate

Install mise, then review mise.toml before trusting it.

git clone https://github.com/feichai0017/oxide-infer.git
cd oxide-infer

mise trust
mise install
USE_MISE=1 make install-website
USE_MISE=1 make check

Run CUDA gates inside the pinned Linux environment:

USE_MISE=1 make cuda-doctor
USE_MISE=1 make cuda-check
USE_MISE=1 make cuda-test
USE_MISE=1 make h20

The environment guide lists the pinned Rust, Node.js, CUDA, and cuda-oxide versions. The H20 guide defines correctness, sanitizer, Graph, and performance gates.

Evidence before claims

Oxide Infer keeps these evidence levels separate:

host reference
  → CUDA correctness
  → lifetime and negative gates
  → CUDA Graph
  → sanitizer
  → matched operator benchmark
  → engine integration
  → serving workload

A lower level does not imply a higher one. Correct output does not prove a speedup. A Graph replay does not prove serving throughput. An adapter hit does not prove end-to-end latency.

Historical Loom Infer records remain immutable. They retain their original provider names, source hashes, and commands.

New Oxide Infer records qualify only the source revision named by each record. See the evidence index.

Roadmap

The roadmap advances through measured vertical slices:

1. Complete the Oxide Infer namespace and framework migration.
2. Requalify current operators on exact H20 source and artifacts.
3. Close the Mistral.rs adapter and define a narrow vLLM C ABI boundary.
4. Compare native M=1 GEMV against Mistral.rs GEMV and cuBLASLt.
5. Expand attention, KV-cache operations, and MLA from engine traces.
6. Add activation, sampling, quantization, grouped GEMM, and MoE paths.
7. Add Blackwell modules only with hardware-backed contracts and evidence.
8. Qualify collectives and distributed integration after single-GPU ownership is stable.

Each milestone has admission, evidence, and stop conditions in the full roadmap.

Contributing

Start with a real engine call site and one measurable contract. Keep one public execution path per operator. Do not add compatibility facades, hidden fallback, or a new crate without a concrete boundary.

Read CONTRIBUTING.md before adding a provider or changing runtime ownership.

License

MIT