TrustformeRS 🦀

A high-performance, memory-safe Rust implementation of Hugging Face Transformers. TrustformeRS brings the power of transformer models to the Rust ecosystem with zero-cost abstractions, fearless concurrency, and deployment flexibility from edge to cloud.

Project Status (alpha): TrustformeRS 0.1.4 (in development, last verified 2026-07-02) is a large Pure-Rust transformer stack — 2,983 Rust files, ~1.4M lines (~1.18M lines of code, via tokei) across 10 crates and 49+ transformer architectures — together with multi-platform packaging (WebAssembly, server REST/gRPC/GraphQL, mobile iOS/Android, and RLHF/DPO training scaffolding).

Honest maturity note: today's compute path is primarily CPU and f32. F16/BF16 are supported as a storage/serialization format but are upcast to f32 for arithmetic (native low-precision kernels are on the roadmap). GPU acceleration is real (CUDA via the Pure-Rust oxicuda backend, Metal via objc2/oxicuda-metal, WebGPU via wgpu) but is currently wired end-to-end only for GPT-2 and RetNet; the remaining backends (ROCm, Vulkan, OpenCL) are feature-gated and experimental, and TPU is a placeholder, not implemented. Several newer architectures are still being completed. See Development Status for the precise maturity of each area.

🚀 Why TrustformeRS?

• 🏎️ Performance: Leverages Rust's zero-cost abstractions, SIMD optimizations, and efficient memory management
• 🔒 Safety: Memory-safe by design with Rust's ownership model - no more segfaults or memory leaks
• 📦 Portability: Deploy anywhere from WebAssembly to embedded devices to GPU clusters
• 🔧 Control: Explicit resource management following SciRS2's Core Usage Policy
• 🤝 Compatibility: Loads Hugging Face model formats directly

📊 Performance (indicative)

⚠️ The figures below are indicative targets on the authors' reference hardware, not the output of an automated benchmark gate, and your results will vary. Reproduce on your own machine with cargo bench (benchmark sources live in benches/). All numbers are CPU f32 — GPU benchmarks are intentionally omitted until GPU coverage extends beyond GPT-2/RetNet.

Reference CPU: Intel i9-12900K. GPU benchmarks will be published once on-device coverage is generalized beyond GPT-2/RetNet (see roadmap).

🏗️ Architecture

TrustformeRS follows a modular workspace structure inspired by Hugging Face Transformers:

trustformers/
├── trustformers-core/      # Core traits and tensor abstractions  (189,922 SLoC, Stable)
├── trustformers-models/    # 49+ model implementations           (185,954 SLoC, Alpha)
├── trustformers-tokenizers/# BPE, WordPiece, SentencePiece       ( 48,701 SLoC, Stable)
├── trustformers-optim/     # 20+ optimizers and LR schedulers    ( 76,662 SLoC, Stable)
├── trustformers-training/  # Distributed training, RLHF/DPO      ( 87,017 SLoC, Stable)
├── trustformers-serve/     # REST/gRPC/GraphQL serving           (331,151 SLoC, Stable)
├── trustformers-wasm/      # WebAssembly + WebGPU deployment     ( 53,361 SLoC, Stable)
├── trustformers-mobile/    # iOS/Android deployment              (125,131 SLoC, Alpha)
├── trustformers-debug/     # Profilers, visualizers, TensorBoard (100,417 SLoC, Alpha)
└── trustformers/           # High-level integration crate        (131,961 SLoC, Alpha)

Total: ~1.33M SLoC across these 10 crates; 2,983 Rust files, ~1.4M lines total (~1.18M lines of code) across the full repository including bindings/examples/tooling (via tokei, 2026-07-01). 100% Pure Rust source (COOLJAPAN Policy) — default-feature builds are C/C++-free for every crate except trustformers-serve (accepted exception: rustls/aws-lc-rs TLS for the HTTP server).

Design Principles

1. Trait-based abstractions: Models, layers, and tokenizers implement common traits for composability
2. Feature-gated backends: Choose between CPU, GPU, or WebAssembly targets
3. Zero-copy model loading: Memory-mapped weights with SafeTensors format
4. Explicit parallelism: You control thread and GPU usage, not the library

🚦 Quick Start

Installation

[dependencies]
trustformers = "0.1.4"

Basic Usage

use trustformers::prelude::*;
use trustformers::{AutoModel, AutoTokenizer, Tensor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load model and tokenizer
    let tokenizer = AutoTokenizer::from_pretrained("bert-base-uncased")?;
    let model = AutoModel::from_pretrained("bert-base-uncased")?;

    // Tokenize input
    let tokenized = tokenizer.encode("Hello, Rust world!")?;

    // AutoModel's `Model` impl is Tensor-in/Tensor-out (uniform across every
    // architecture it wraps), so wrap the token IDs as a Tensor before running
    // inference.
    let ids: Vec<f32> = tokenized.input_ids.iter().map(|&id| id as f32).collect();
    let len = ids.len();
    let inputs = Tensor::from_vec(ids, &[len])?;

    // Run inference
    let outputs = model.forward(inputs)?;

    println!("Output shape: {:?}", outputs.shape());
    Ok(())
}

(For architecture-specific outputs with named fields — e.g. BertModelOutput { last_hidden_state, pooler_output } — construct the concrete model type directly, such as trustformers::BertModel, whose forward takes the TokenizedInput from tokenizer.encode(..) straight through.)

Pipeline API

use trustformers::pipeline;

// All pipelines are fully implemented and ready to use!
// `pipeline(task, model, options)` — pass `None, None` for the defaults.
let classifier = pipeline("sentiment-analysis", None, None)?;
let result = classifier.__call__("I love writing Rust code!".to_string())?;
// Output: PipelineOutput::Classification([ClassificationOutput { label: "POSITIVE", score: 0.999 }])

// Also available:
// - text-generation
// - token-classification (NER)
// - question-answering
// - fill-mask
// - summarization
// - translation

🏛️ Model Zoo

Currently Supported (49+ architectures!)

Encoder Models

Decoder Models

Encoder-Decoder Models

Vision & Multimodal Models

State-Space & Linear Attention Models

⚡ Performance Features

TrustformeRS includes state-of-the-art optimizations not mentioned in typical documentation:

• FlashAttention & FlashAttention-2: O(N) memory complexity for attention
• PagedAttention: Efficient KV cache management for long sequences
• INT8/INT4 Quantization: GPTQ and AWQ quantization methods
• Mixed Precision (partial): FP16/BF16 weight storage + casting/loss-scaling utilities; core arithmetic currently upcasts to FP32 (native FP16/BF16 compute kernels are on the roadmap)
• ZeRO Optimization: All 3 stages for distributed training
• SIMD Operations: Leveraging SciRS2 for vectorized computations
• Tensor Parallelism: Split large models across multiple GPUs
• Gradient Checkpointing: Trade compute for memory efficiency

🚀 Deployment Options

TrustformeRS supports multiple deployment targets:

• WebAssembly: Browser deployment (trustformers-wasm, Stable)

  • WebGPU acceleration support
  • JavaScript/TypeScript bindings
  • React/Vue component-ready

• Server: Production-ready API serving (trustformers-serve, Stable)

  • REST, gRPC, and GraphQL endpoints
  • Dynamic batching with Kubernetes deployment manifests
  • Docker containers and auto-scaling support

• Training: Full training infrastructure (trustformers-training, Stable)

  • RLHF and DPO training support
  • Distributed training with ZeRO optimization
  • Mixed precision (FP16/BF16)

• Mobile: Native mobile deployment (trustformers-mobile, Alpha)

  • iOS framework with Core ML and Metal acceleration
  • Android library with NNAPI and Vulkan support
  • React Native, Flutter, and Unity integrations

• Edge: Export to optimized formats

  • ONNX export/import
  • GGUF format support
  • Quantized models (INT8/INT4, GPTQ, AWQ) for embedded devices

🛠️ Advanced Usage

Custom Model Implementation

use trustformers_core::{Model, Layer, Config};
use trustformers_core::layers::Embedding; // real building block
use trustformers_core::traits::TokenizedInput;

// `TransformerEncoder`, `Pooler`, `MyConfig`, and `ModelOutput` are illustrative
// types you define yourself, typically composed from real trustformers_core
// building blocks such as `MultiHeadAttention`, `LayerNorm`, and `FeedForward`
// (all in `trustformers_core::layers`).
struct MyTransformer {
    embeddings: Embedding,
    encoder: TransformerEncoder,
    pooler: Pooler,
}

impl Model for MyTransformer {
    type Config = MyConfig;
    type Input = TokenizedInput;
    type Output = ModelOutput;

    fn forward(&self, input: Self::Input) -> Result<Self::Output> {
        let hidden_states = self.embeddings.forward(input.input_ids)?;
        let encoded = self.encoder.forward(hidden_states)?;
        let pooled = self.pooler.forward(encoded.clone())?;

        Ok(ModelOutput { hidden_states: encoded, pooled_output: pooled })
    }
}

GPU Acceleration

GPU backends are real (CUDA via the Pure-Rust oxicuda backend, Metal via objc2/oxicuda-metal, WebGPU via wgpu) but are currently wired end-to-end only for GPT-2 and RetNet. Enable the matching feature flag (metal on macOS, cuda on Linux/Windows) and build a supported model on a GPU Device; its forward then runs the linear/attention path on-device with a persistent KV cache.

CUDA runtime-verified (2026-07-01): the CUDA backend was migrated from cudarc to the Pure-Rust oxicuda (oxicuda-blas/-dnn/-memory/-driver) — the cuda feature now pulls in oxicuda instead of cudarc (cuda-oxicuda is kept only as a deprecated alias for cuda). 12 CPU↔CUDA golden-parity tests (GEMM, GELU, LayerNorm, causal softmax, RoPE — both host and GPU-resident paths, plus cached-weight GEMM) prove the backend is numerically correct against the CPU reference, runtime-verified 12/12 passing on a real NVIDIA RTX A4000 (CUDA 12.0). The GPU-resident CUDA transformer layer (LayerNorm→QKV→bias→RoPE→causal-softmax attention→proj→residual, chained via cached device buffers with no host round-trips) replaces the previous CPU-fallback placeholder. Metal compute (matmul + resident attention) similarly migrated to oxicuda-metal, dropping the earlier scirs2-core MPS dependency.

// Cargo.toml: trustformers-core = { version = "0.1", features = ["metal"] }  // or "cuda"
use trustformers_core::Device;
use trustformers_models::gpt2::{Gpt2Config, Gpt2Model};

// Construct a supported model (e.g. GPT-2) and move its weights to a GPU device.
let device = Device::Metal(0); // or Device::CUDA(0)
let mut model = Gpt2Model::new_with_device(Gpt2Config::default(), device)?;
model.weights_to_gpu(&device)?;        // Metal (use `weights_to_gpu_cuda` on CUDA)
let outputs = model.forward(inputs)?;  // attention + linear run on-device

A generic model.to_gpu() covering all 49+ architectures is not yet available — broader coverage is tracked under Development Status. For unsupported models, inference currently runs on CPU (f32).

WebAssembly Deployment

# Build for WASM
cargo build --target wasm32-unknown-unknown --features wasm

# Use in JavaScript (real wasm-bindgen exports: WasmTokenizer/BertModelWasm,
# constructed from a config rather than a HuggingFace Hub name)
import init, { WasmTokenizer, TokenizerType, BertModelWasm, BertConfig } from './trustformers_wasm.js';

await init();
const tokenizer = new WasmTokenizer(TokenizerType.WordPiece);
const model = new BertModelWasm(new BertConfig());

const ids = tokenizer.encode("Hello, Rust world!", true); // -> token ID array
const hiddenStates = model.forward(ids);

🔄 Migration from Python

TrustformeRS maintains API similarity with Hugging Face Transformers for easy migration:

from transformers import (
    AutoModel, 
    AutoTokenizer
)

tokenizer = AutoTokenizer.from_pretrained(
    "bert-base-uncased"
)
model = AutoModel.from_pretrained(
    "bert-base-uncased"
)

inputs = tokenizer(
    "Hello world!", 
    return_tensors="pt"
)
outputs = model(**inputs)

use trustformers::{
    AutoModel, 
    AutoTokenizer,
    Tensor,
};

let tokenizer = AutoTokenizer::from_pretrained(
    "bert-base-uncased"
)?;
let model = AutoModel::from_pretrained(
    "bert-base-uncased"
)?;

let tokenized = tokenizer.encode(
    "Hello world!"
)?;
// AutoModel is Tensor-in/Tensor-out; wrap token IDs first.
let ids: Vec<f32> = tokenized.input_ids
    .iter().map(|&i| i as f32).collect();
let len = ids.len();
let inputs = Tensor::from_vec(ids, &[len])?;
let outputs = model.forward(inputs)?;

🎯 Development Status

Completed Features (v0.1.4 - 2026-07-01)

• CUDA backend migrated to the Pure-Rust oxicuda (COOLJAPAN Pure-Rust policy), entirely replacing cudarc (cuda now pulls oxicuda-blas/-dnn/-memory/-driver; cuda-oxicuda kept as a deprecated alias). GPU-resident matmul_gpu_to_gpu confirmed genuinely zero-copy (cached DeviceBuffer, no host round-trip); the cuda feature now propagates through trustformers-models and the trustformers umbrella crate.
• 12 CPU↔CUDA golden-parity tests (GEMM, GELU, LayerNorm, causal softmax, RoPE — host and GPU-resident paths, plus cached-weight GEMM), runtime-verified 12/12 passing on a real NVIDIA RTX A4000 (CUDA 12.0).
• Real GPU-resident CUDA transformer layer: pre-norm causal self-attention (LayerNorm→QKV→bias→RoPE→causal-softmax attention→proj→residual) chained via cached device buffers with no host round-trips, replacing the previous CPU-fallback placeholder.
• Metal GPU compute migrated to oxicuda-metal (Pure Rust), dropping the scirs2-core MPS dependency entirely; GPU-resident matmul is zero-copy; GPT-2's feed-forward now uses a single fused matmul+bias+GELU Metal kernel (one GPU dispatch instead of three). Verified on Apple Silicon.
• Real PyRwkvModel / PyMambaModel Python classes: AutoModel now loads RWKV/Mamba checkpoints correctly instead of silently falling back to BERT (the Python binding layer was also re-enabled and modernized to PyO3 0.28).
• WebGPU device/queue initialization in the WebAssembly compute backend (navigator.gpu → adapter → device), falling back to CPU when no adapter is available.
• Eliminated production-code unwrap()/expect() across the entire workspace (lock-poison recovery, proper Result propagation, documented infallible invariants only where truly load-bearing); no public API changes, all tests pass unchanged.
• Default feature trees are Pure-Rust (C/C++-free) for every crate except trustformers-serve (HTTP server; keeps rustls/aws-lc-rs TLS, accepted exception). Networking (HuggingFace Hub downloads, remote leaderboard), debug visualization, and serve's AWS-Lambda/Swagger-UI adapters are now opt-in behind features (hub, remote-leaderboard, visual, lambda, swagger-ui); switched the tokenizer regex backend to pure-Rust fancy-regex.
• Restored gRPC proto compilation and serving (migrated build.rs to the tonic 0.14 split tonic-build/tonic-prost-build API).
• Removed the legacy cudarc-based CUDA backend and an orphaned, never-mounted rope/mod.rs module (~1,693 lines whose RoPE convention was inconsistent with the live, now parity-tested kernel).
• Full workspace verification (2026-07-01): cargo nextest run --workspace --all-features = 18,102 passed, 0 failed (119 skipped, ~565s) · cargo clippy --workspace --all-features --all-targets -- -D warnings = 0 warnings/errors · cargo doc --workspace --all-features --no-deps (RUSTDOCFLAGS="-D warnings") = 0 warnings · cargo fmt --all -- --check = clean.

Completed Features (v0.1.3 - 2026-06-24)

• Gorilla time-series compression + historical-data engine (trustformers-serve): delta-of-delta timestamp encoding plus XOR float encoding for metric series, with a real lifecycle/archival/query engine (expiry cleanup, archive/retrieve, cached query execution) replacing prior stubs
• Real concurrency-detector analytics (trustformers-serve): working cycle/deadlock, thread, lock, and conflict detection with pattern, sharing, and risk-assessment scoring across all eight detector modules
• Interpretability tools wired in (trustformers-debug): real SHAP, LIME, and Integrated-Gradients feature attribution exposed as InterpretabilityAnalyzer / InterpretabilityConfig / InterpretabilityReport
• GlobalMemoryPool / ZeroCopyTensorView / GlobalProfiler (trustformers): a thread-safe aligned allocator, a bounds-checked borrowed f32 tensor view, and a per-session profiler operation API, all re-exported from the crate root
• Real .xlsx export (trustformers-debug): emits a valid OOXML workbook package via oxiarc-archive (Pure Rust), replacing the previous CSV-with-.xlsx-extension placeholder
• Real SHA-256 on the HuggingFace upload path (via sha2), replacing a non-cryptographic XOR-fold stub
• Reliability fixes: clap -c short-flag collisions that panicked the load_test / message_queue_cli binaries are fixed; Miri-verified memory-pool fixes for heap mis-layout deallocation (undefined behavior) and a block-reuse leak

Completed Features (v0.1.2 - 2026-06-20)

• 49+ transformer architectures for CPU inference (BERT, RoBERTa, ALBERT, DistilBERT, ELECTRA, DeBERTa, GPT-2, GPT-Neo, GPT-J, GPT-NeoX, LLaMA, Mistral, Gemma, Qwen, Phi-3, Falcon, StableLM, T5, ViT, CLIP, BLIP-2, LLaVA, DALL-E, Flamingo, Mamba, RWKV, S4, Falcon2, Gemma2, Granite, Hyena, InternLM2, Jamba, Jamba2, Linformer, LLaMA3.2, Mamba2, Nemotron, Performer, Phi4, Qwen2.5, RetNet, SD3, StarCoder2, Whisper, xLSTM, Yi). Maturity varies — the BERT/GPT-2/LLaMA/T5/ViT families are the most exercised; several newer or experimental architectures are alpha-quality and still being completed for full numerical parity with the reference implementations.
• All major NLP pipelines fully implemented (text-generation, classification, QA, NER, fill-mask, summarization, translation)
• Complete training infrastructure with distributed training, ZeRO optimization, mixed precision, RLHF and DPO support
• Mobile deployment with iOS (Core ML, Metal) and Android (NNAPI, Vulkan) support
• WebAssembly deployment with WebGPU acceleration
• REST/gRPC/GraphQL APIs with dynamic batching, Kubernetes deployment, and autoscaling
• Safety filtering pipeline with configurable content moderation
• Advanced optimizations: FlashAttention, PagedAttention, quantization (INT8/INT4/GPTQ/AWQ)
• GPU backends: CUDA (cudarc) and Metal (MPS) wired into GPT-2/RetNet forward paths; WebGPU/Vulkan/OpenCL/ROCm present as feature-gated backends (experimental, not yet wired into model forward)
• AutoModel/AutoTokenizer system with HuggingFace Hub integration
• Large test suite: 18,008 tests pass via cargo nextest run --all-features (119 skipped) across the full workspace, verified locally on 2026-06-24
• Debugging tools: Profilers, visualizers, interactive debugging, TensorBoard integration
• 100% Pure Rust (COOLJAPAN Policy) - ~1,408,134 SLoC across 10 crates

Future Enhancements

High Priority

• Broader GPU model coverage: extend the oxicuda/oxicuda-metal device-resident forward path beyond GPT-2/RetNet to more architectures (superseded the earlier scirs2-core MPSGraph plan — Metal now runs on oxicuda-metal directly, no longer blocked on scirs2-core)
• More quantization methods: Enhanced GGUF format, AutoGPTQ improvements
• Additional vision transformer variants: ViT-Huge, DeiT, Swin

Performance

• Fused CUDA megakernel: LayerNorm+QKV+RoPE+Attention+Proj+Residual as a single oxicuda-dnn kernel (today's oxicuda path executes ops individually — correct, but not fused)
• Streaming inference: Real-time token streaming for all generation pipelines

Documentation

• Comprehensive guides: Model implementation, deployment, optimization tuning
• Cookbook: Common patterns and best practices

🤝 Contributing

We welcome contributions! See our Contributing Guide for details.

Adding a New Model

1. Create a new module in trustformers-models/src/
2. Implement the Config, Model, and task-specific heads
3. Add tests comparing outputs with Hugging Face
4. Submit a PR with benchmarks

Performance Contributions

• Profile with cargo-flamegraph
• Benchmark with criterion
• Consider SIMD optimizations for hot paths
• Ensure thread-safety for concurrent use

📈 Benchmarks

Run benchmarks with:

cargo bench --all-features

View detailed results in target/criterion/report/index.html

🛡️ Safety and Security

• No unsafe code in public APIs (only in carefully reviewed hot paths)
• All models are Send + Sync for safe concurrent use
• Fuzzing tests for tokenizers
• Memory usage bounds for OOM prevention

📚 Documentation

• API Documentation
• Architecture Guide
• Performance Tuning
• Model Implementation Guide

🙏 Acknowledgments

• Inspired by Hugging Face Transformers
• Built on SciRS2 for scientific computing
• Tokenizers from Hugging Face Tokenizers
• Community contributions and feedback

Sponsorship

TrustFormers is developed and maintained by COOLJAPAN OU (Team Kitasan).

If you find TrustFormers useful, please consider sponsoring the project to support continued development of the Pure Rust ecosystem.

https://github.com/sponsors/cool-japan

Your sponsorship helps us:

• Maintain and improve the COOLJAPAN ecosystem
• Keep the entire ecosystem (OxiBLAS, OxiFFT, SciRS2, etc.) 100% Pure Rust
• Provide long-term support and security updates

📄 License

Licensed under Apache License, Version 2.0 (LICENSE).

🌟 Star History

Built with 🦀 and ❤️ by COOLJAPAN OU (Team KitaSan)