memra

August 24, 2026 · View on GitHub

Rust + CUDA inference engine tuned separately for RTX PRO 6000 Blackwell and RTX 5090, with OpenAI-compatible serving and model-specific correctness gates.

Install · Models · Serving · Performance · Hosted API

Quick start

Prebuilt binaries require Linux x86_64, an NVIDIA driver with CUDA 13 runtime support, and the CUDA runtime libraries. The installer selects the GPU build, verifies its checksum, and does not require nvcc.

curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
kernel-check                  # expect: ALL GREEN

Start a server with a supported public checkpoint. The first run downloads and caches the model:

MEMRA_MODELS="q38=hf:Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp" \
memra-server

The server listens on 127.0.0.1:8080. Send a streaming chat completion from another terminal:

curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "q38",
    "messages": [{"role": "user", "content": "Explain KV caching in one sentence."}],
    "max_tokens": 128,
    "stream": true
  }'

Use the cookbook for qualified model-and-card configurations. See installation for source builds and platform requirements.

Project spirit

Memra is narrow on purpose. The best execution path depends on the model, checkpoint, hardware, and workload, so Memra measures those combinations separately instead of promising that one loader or one default supports everything. A fast path is promoted only when its own correctness and performance evidence is current.

The project favors explicit support, reproducible receipts, and useful failures over silent fallbacks. Issues, model requests, hardware reports, and evidence-backed pull requests are welcome. Look elsewhere if you need broad hardware coverage, a large architecture catalog, or a Python library embedded in your application.

Choose a path

By model: Qwen3.8 27B · Qwen3.5 9B · Qwen3.6 27B · Qwen3.6 35B-A3B · Qwen-AgentWorld 35B-A3B · Ornith 1.5 35B-A3B · Ornith 1.0 9B · Ornith 1.0 35B · Gemma 4 12B · Gemma 4 26B-A4B · Gemma 4 31B · Gemma 4 E4B · Step 3.7 Flash · DeepSeek V4 Flash

By hardware: RTX PRO 6000 Blackwell · RTX 5090 / 50-series · H100 · Ada · B200

By workload: interactive agents · long-context and prefix reuse · batch throughput · multimodal · large models on multiple GPUs

Documentation

Document	Use it for
Installation	Prebuilt requirements, source builds, architecture selection
Model cards	Best starting path for each supported model
Hardware cards	Recommended use by GPU target
Workload cards	Recommended use by request shape
Cookbook	Copy-paste model and card configurations
Models	Supported checkpoints, formats, drafters, and hardware
Serving	HTTP contract, caching, auth, admission, multi-GPU, operations
API surfaces	Anthropic Messages and OpenAI Responses compatibility
Performance	Measurements, methodology, rigs, and receipts
Flags	Audited environment-variable reference
Testing	Correctness gates and evidence requirements
Architecture	Runtime structure and Blackwell implementation ledger
Decisions	Adopted and rejected design choices with evidence
Releases	Changelog and release artifacts

memra

Quick start

Project spirit

Choose a path

Documentation

Issues and requests

Contributing

License