OxiGAF

August 28, 2026 · View on GitHub

Pure Rust Gaussian Avatar Reconstruction from monocular videos via multi-view diffusion.

Implements the methods from GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-View Diffusion entirely in the Rust ecosystem.

What's in v0.1.2

Gradient-Correctness Fixes (read before upgrading)

  • Backward rasterizer shaders fixed: rasterize_bwd.wgsl could accumulate a tile's gradient onto the wrong Gaussian (a WGSL workgroup-uniformity bug around workgroupBarrier()), and preprocess_bwd.wgsl omitted the position gradient through view-dependent spherical-harmonics color for every sh_degree >= 1 model. Both affected every training run on 0.1.1 — retraining is the only remedy. See CHANGELOG.md's [0.1.2] migration notes for the full detail.
  • Trainer::compute_gradients no longer hardcodes an L2 photometric loss — LossConfig's w_l1/w_ssim/w_ms_ssim weights previously only affected the logged loss, never what the optimizer actually descended. Position/scale/opacity regularization losses and SDS (score-distillation) training likewise went from logged-only to actually producing gradients.

New Capabilities (v0.1.2)

  • Pure-Rust PyTorch checkpoint ingest: oxigaf-bridge now parses .pt/ .pkl checkpoints directly (convert_pytorch_checkpoint, convert_flame_model) — no Python, no PyTorch, no torch.load needed, and the old scripts/convert_*.py fallbacks are deprecated in its favor.
  • Spec-conformant glTF 2.0 export: a new oxigaf_render::gltf module consolidates what were three independently-written, mutually-incompatible glTF writers in the workspace into one that actually satisfies the spec (per-accessor buffer views, mandatory min/max).
  • Gaussian pruning, LR schedules, gradient clipping as config: oxigaf-trainer gained pruning::GaussianPruner, a 6-variant LrScheduleConfig, and a GradientClipConfig — all genuinely wired into Trainer::train_step via TrainingConfig, not just standalone APIs.
  • A real meta-learning avatar model: meta_learning_avatar:: GaussianAvatarModel is the first MetaModel implementation over an actual Gaussian avatar (the prior LinearModel was a disconnected toy).
  • GPU-side pass profiling: oxigaf_render::profiler::GpuTimestampProfiler (backed by wgpu::Features::TIMESTAMP_QUERY), alongside device-limit validation so an under-provisioned GPU fails fast with a clear error instead of an opaque pipeline-validation panic.

Also Fixed

  • PLY files with sh_degree >= 1 written before this release load with permuted higher-order SH coefficients (f_rest_* property order is now channel-major, matching the reference 3DGS Python convention) — re-export any you care about.
  • macOS: the default asset cache directory moved to ~/Library/Caches/oxigaf (setup/doctor/cache previously disagreed on where it lived).
  • oxigaf::pipeline::export/render_from_file were previously no-op stubs despite their own doc comments describing real behavior — now genuinely load, convert, render, and save.
  • Remaining C dependencies removed from oxigaf-cli's HTTP stack (OpenSSL, then ring) in favor of a pure-Rust ureq/rustls/RustCrypto stack.

See CHANGELOG.md for the complete, verified list (breaking signature changes, deprecated APIs, and everything above with exact type/function names).

512×512 Multi-View Generation (v0.1.0)

  • Latent Upsampler: 32×32 → 64×64 latent upsampling for 512×512 output resolution
  • IP-Adapter: Identity-preserving image conditioning for consistent face/object generation
  • Classifier-Free Guidance: Quality improvement with configurable guidance scale (1.0-20.0)
  • Multi-view UNet: Cross-view attention for geometric consistency across views
  • Camera Conditioning: Explicit camera pose embeddings for view-aware generation

Extended Head Model (v0.1.1)

  • Avatar Rigging & Expressions: AvatarRig, GazeController, expression animation with FACS AU coefficients, phoneme-driven animation, emotion recognition
  • Mesh Processing Suite: Mesh repair, smoothing, Loop/Catmull-Clark subdivision, morphing, geodesic distance, spectral analysis
  • UV & Texture Pipeline: UV parameterisation, texture baking, face atlas generation, albedo maps, SH lighting
  • Motion & Deformation: Timeline, warp field, shape retargeting, dynamic landmark tracking

Comprehensive Rendering Pipeline (v0.1.1)

  • Post-Processing: SSAO, bloom, depth-of-field, motion blur, HDR tone mapping, film grain, chromatic aberration, TAA
  • Scene Composition: Render graph, image compositor, silhouette extraction, background synthesis
  • Volumetric Rendering: Ray types, camera model, volume grid, ray-march result traits
  • Stereo Output: Side-by-side and top-bottom stereo rendering

Extended Training & Diffusion (v0.1.1)

  • Sampler Suite: DDPM, adaptive sampling, consistency model, flow matching, guidance rescaling
  • LoRA & ControlNet Adapters: Parameter-efficient fine-tuning and conditioning adapters
  • Curriculum Learning: Progressive training, few-shot adaptation, meta-learning (MAML), continual learning
  • Gradient Tools: Gradient surgery, OHEM, anomaly detection, activation maps

Expanded CLI Toolset (v0.1.1)

  • Export Suite: PLY, glTF, mesh, point cloud, video, and animation sequence export
  • Analysis Tools: Scene analyser, model inspector, diff tool, model comparison, quality checker
  • Scene Operations: Scene merging, optimiser, streaming, Gaussian filter and deduplication
  • Visualisation: Arcball camera controller, LOD generator, camera path editor, live dashboard

Quality & Performance

  • 100% Pure Rust: Zero C/Fortran dependencies (COOLJAPAN compliant)
  • Comprehensive test suite: validation across all crates — see CHANGELOG.md / CI for current counts (a specific number here would only go stale)
  • Production Ready: Zero unwrap(), feature-gated dependencies, splitrs-based file-size policy (target: under 2000 lines per file)

Workspace Structure

CrateTypeDescription
oxigaf-flamelibFLAME parametric head model (LBS, normal maps, safetensors I/O, video sequences)
oxigaf-diffusionlibMulti-view diffusion with IP-Adapter, upsampling, and CFG (candle)
oxigaf-renderlibDifferentiable 3D Gaussian Splatting rasterizer with CPU reference (wgpu)
oxigaf-trainerlibOptimization pipeline with gradient verification and FLAME binding backward
oxigaf-bridgelibPyTorch ↔ OxiGAF weight conversion and layer mapping utilities. Standalone library — add it as a dependency in your own project; it is not currently wired into the oxigaf CLI binary (the CLI's convert subcommand only handles FLAME .pkl/.npz).
oxigaflibMeta crate — unified re-export of all sub-crates
oxigaf-clibinCLI binary (oxigaf command)

Quick Start

# Build the workspace
cargo build --workspace

# Run the CLI
cargo run -p oxigaf-cli -- --help

# Run tests
cargo test --workspace

Feature Flags

OxiGAF supports various feature flags for platform-specific optimizations:

Platform-Independent Features

FeatureDescription
simdSIMD optimizations for FLAME model (requires nightly Rust)
parallelParallel processing with rayon
flash_attentionMemory-efficient attention mechanism
mixed_precisionFP16/BF16 inference
gpu_debugGPU validation layers and debug markers

GPU / BLAS Backends

OxiGAF does not define its own cuda / metal / accelerate feature flags — no crate in this workspace declares them. GPU/BLAS acceleration for the candle-based crates (oxigaf-diffusion, oxigaf-trainer) is configured by depending on the oxicandle-core fork directly with its own features, e.g. in a downstream Cargo.toml:

candle-core = { package = "oxicandle-core", version = "0.11.0", features = ["metal"] }      # macOS GPU
candle-core = { package = "oxicandle-core", version = "0.11.0", features = ["accelerate"] } # macOS BLAS
candle-core = { package = "oxicandle-core", version = "0.11.0", features = ["cuda"] }        # NVIDIA GPU

The 3D Gaussian Splatting rasterizer (oxigaf-render) always uses wgpu, which auto-selects Metal / Vulkan / DirectX / GL at runtime — no feature flag needed there.

Building Documentation

cargo doc --no-deps --features "simd,parallel,flash_attention,mixed_precision,gpu_debug"

# Enforce warnings as errors
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --features "simd,parallel,flash_attention,mixed_precision,gpu_debug"

Building with Features

# CPU-only build (all platforms)
cargo build --release --features "simd,parallel,flash_attention"

See "GPU / BLAS Backends" above to additionally enable oxicandle-core's metal / accelerate / cuda features for oxigaf-diffusion/oxigaf-trainer.

FLAME Model Setup

OxiGAF supports both legacy NPY and modern Safetensors formats for FLAME models.

Safetensors format is supported for runtime loading/saving. The PyTorch conversion step is pure-Rust as of v0.1.2 (previously Python-only).

  1. Download the FLAME 2023 model from https://flame.is.tue.mpg.de/
  2. Convert the PyTorch checkpoint to safetensors — pure Rust, no Python, PyTorch, or torch.load needed (partitions into unet/vae/clip/ other; --precision fp16 matches the old Python script's behaviour, which always forced FP16 — omit it to keep each tensor's original dtype): 
    cargo run -p oxigaf-bridge --example convert_pytorch -- \
  --checkpoint path/to/checkpoint.pt --output-dir output_dir/ --precision fp16
    A Python fallback is still available if you prefer it: python scripts/convert_weights.py path/to/checkpoint.pt output_dir/

Option 2: NPY (Legacy)

  1. Download the FLAME 2023 model from https://flame.is.tue.mpg.de/
  2. Convert to .npy format — pure Rust: 
    cargo run -p oxigaf-bridge --example convert_flame_pkl -- \
  --model path/to/FLAME2023.pkl --output-dir output_dir/
    A Python fallback is still available if you prefer it: python scripts/convert_flame.py path/to/FLAME2023.pkl output_dir/

Usage Examples

Multi-View Diffusion Pipeline (v0.1.1)

use oxigaf_diffusion::{MultiViewDiffusionPipeline, DiffusionConfig};
use candle_core::Device;
use std::path::Path;

// Configure multi-view generation with classifier-free guidance
let config = DiffusionConfig {
    num_views: 4,
    guidance_scale: 7.5,
    num_inference_steps: 50,
    ..Default::default()
};

// Load the complete pipeline
let device = Device::cuda_if_available(0)?;
let pipeline = MultiViewDiffusionPipeline::load(
    config,
    Path::new("weights/"),
    &device,
)?;

// Generate multi-view images with camera conditioning
let output = pipeline.generate(&input_image, &camera_poses)?;

Video Sequence Processing (v0.1.1)

use oxigaf_flame::{FlameSequence, FlameParams};
use std::path::Path;

// Load video sequence with LRU caching
let mut sequence = FlameSequence::from_json(Path::new("sequence.json"))?;

// Access frames with automatic caching
let frame_42 = sequence.get_frame(42)?;

// Interpolate between frames
let interpolated = sequence.interpolate(42.5)?;

Safetensors I/O (v0.1.1)

use oxigaf_flame::{load_flame_model_safetensors, save_flame_model_safetensors};
use std::path::Path;

// Load FLAME model from safetensors
let model = load_flame_model_safetensors(Path::new("flame_model.safetensors"))?;

// Save to safetensors (preserves metadata)
save_flame_model_safetensors(&model, Path::new("output.safetensors"))?;

PyTorch Weight Conversion

oxigaf-bridge is a standalone library crate — add oxigaf-bridge to your own Cargo.toml to use it (cargo add oxigaf-bridge); it is not exposed through the oxigaf CLI binary.

use oxigaf_bridge::LayerMapping;

// Create layer mapping for weight conversion
let mut mapping = LayerMapping::new();

// Add custom layer name mappings
mapping.add_custom_mapping(
    "pytorch.layer.weight".to_string(),
    "oxigaf_module_weight".to_string(),
);

// Convert PyTorch layer names to OxiGAF format
let oxigaf_name = mapping.pytorch_to_oxigaf("unet.down_blocks.0.conv.weight")?;
// Result: "down_blocks.0.conv.weight" (dot-separated — VarBuilder-loadable;
// see crates/oxigaf-bridge/README.md for the 0.1.2 naming migration note)

Documentation

  • Design Documents - Original architecture and design plans with implementation status
  • Crate TODOs - Current implementation status in crates/*/TODO.md files
  • Individual Crate READMEs - API documentation in crates/*/README.md

For new contributors:

  1. Start with docs/design/IMPLEMENTATION_PLAN.md for the big picture
  2. Check module-specific plans in docs/design/
  3. Review current status in the corresponding crates/*/TODO.md file

Sponsorship

OxiGAF is developed and maintained by COOLJAPAN OU (Team Kitasan).

If you find OxiGAF useful, please consider sponsoring the project to support continued development of the Pure Rust ecosystem.

Sponsor

https://github.com/sponsors/cool-japan

Your sponsorship helps us:

  • Maintain and improve the COOLJAPAN ecosystem
  • Keep the entire ecosystem (OxiBLAS, OxiFFT, SciRS2, etc.) 100% Pure Rust
  • Provide long-term support and security updates

License

Apache-2.0