• Multiple Output Types: Support for f32, f16, bf16, q8_0, tq1_0, tq2_0, and auto-detection
• 30+ Quantization Methods: From 1-bit to 32-bit precision with K-quant and IQ variants
• Importance Matrix Quantization: Enhanced precision with IQ1, IQ2, IQ3, IQ4 methods
• Remote Mode (Experimental): Convert models without downloading full weights to disk
• Model Splitting: Split large models into manageable chunks with K/M/G size units
• Hardware Acceleration Detection: Automatically detects CUDA, Metal, OpenCL, Vulkan, ROCm
• Hugging Face Integration: Direct download from and upload to Hugging Face repositories
• README Generation: Automatically creates documentation for your quantized models
• Dry Run Mode: Preview split plans before writing any files

The full command syntax is:

python -m webscout.Extra.gguf convert [OPTIONS]

Choose the base output precision before quantization:

# Use bfloat16 (good for models trained with bf16)
python -m webscout.Extra.gguf convert -m "organization/model-name" -o "bf16"

# Use auto-detection based on model's tensor types
python -m webscout.Extra.gguf convert -m "organization/model-name" -o "auto"

# Use 8-bit quantization directly
python -m webscout.Extra.gguf convert -m "organization/model-name" -o "q8_0"

Apply multiple quantization methods at once:

python -m webscout.Extra.gguf convert -m "organization/model-name" -q "q4_k_m,q5_k_m"

This will create two versions of the model with different quantization methods.

Convert models without downloading the full weights to disk:

python -m webscout.Extra.gguf convert -m "organization/model-name" --remote

This downloads only config and tokenizer files, streaming tensor data directly from HuggingFace. Useful for systems with limited disk space.

Convert and upload the model to your Hugging Face account:

python -m webscout.Extra.gguf convert -m "organization/model-name" -u "your-username" -t "your-token"

This will create a new repository in your account named model-name-GGUF containing the converted model.

Use importance matrix for more efficient quantization:

python -m webscout.Extra.gguf convert -m "organization/model-name" -i --train-data "train_data.txt" -q "iq4_nl"

Importance matrix helps focus more bits on weights that matter most for the model's performance.

Split large models for easier distribution:

# Split by number of tensors
python -m webscout.Extra.gguf convert -m "organization/model-name" -s --split-max-tensors 256

# Split by file size (supports K, M, G units)
python -m webscout.Extra.gguf convert -m "organization/model-name" -s --split-max-size "2G"

# Preview split plan without writing files
python -m webscout.Extra.gguf convert -m "organization/model-name" -s --split-max-size "2G" --dry-run

This is useful for very large models that may be difficult to distribute as a single file.

These methods require --use-imatrix and optionally --train-data:

For example, a 7B parameter model requires:

• f16: ~14GB RAM
• q4_k_m: ~3.5GB RAM

The converter automatically detects and utilizes:

• CUDA for NVIDIA GPUs
• Metal for Apple Silicon and AMD GPUs on macOS
• OpenCL for cross-platform GPU acceleration
• Vulkan for cross-platform GPU acceleration
• ROCm for AMD GPUs on Linux

If no acceleration is available, the converter will use CPU-only mode.

python -m webscout.Extra.gguf convert \
    -m "mistralai/Mistral-7B-Instruct-v0.2" \
    -q "q4_k_m" \
    -o "f16" \
    -u "your-username" \
    -t "your-token"

This will convert Mistral-7B to q4_k_m quantization and upload it to your Hugging Face account.

python -m webscout.Extra.gguf convert \
    -m "meta-llama/Meta-Llama-3-8B" \
    -o "bf16" \
    -q "q4_k_m,q5_k_m,q6_k"

This will create three versions of the model starting from bf16 base.

python -m webscout.Extra.gguf convert \
    -m "mistralai/Mistral-7B-Instruct-v0.2" \
    -q "iq4_nl,iq3_m" \
    -i \
    --train-data "my_training_data.txt"

This will create two IQ-quantized versions using importance matrix for better quality.

python -m webscout.Extra.gguf convert \
    -m "meta-llama/Meta-Llama-3-8B" \
    --remote \
    -q "q4_k_m"

Convert without downloading full model weights (streams from HuggingFace).

# Preview the split plan first
python -m webscout.Extra.gguf convert \
    -m "meta-llama/Llama-2-70b-chat-hf" \
    -q "q4_k_m" \
    -s \
    --split-max-size "4G" \
    --dry-run

# Then actually perform the split
python -m webscout.Extra.gguf convert \
    -m "meta-llama/Llama-2-70b-chat-hf" \
    -q "q4_k_m" \
    -s \
    --split-max-size "4G"

This will split the large 70B model into multiple files, each no larger than 4GB.

Error: Missing required dependencies: git, cmake

Solution: Install the required system dependencies:

• Ubuntu/Debian: sudo apt install git cmake python3-dev build-essential
• macOS: brew install git cmake
• Windows: Install Git and CMake from their respective websites

For hardware acceleration, install relevant drivers (CUDA, ROCm, etc.)

Error: CUDA out of memory

Solutions:

1. Try a lower precision quantization method: q3_k_s or q2_k
2. Enable model splitting with -s
3. Use --no-lazy to disable lazy evaluation (uses more RAM but may help)
4. Try remote mode with --remote to reduce disk I/O
5. Increase your system's swap space/virtual memory

Error: Invalid output type: xxx

Solution: Use a valid output type:

• f32 - Full 32-bit precision
• f16 - Half 16-bit precision
• bf16 - bfloat16 precision
• q8_0 - 8-bit quantization
• tq1_0 - 1-bit ternary
• tq2_0 - 2-bit ternary
• auto - Auto-detect

Error: Failed to download model

Solutions:

1. Check your internet connection
2. Verify you have access to the model on Hugging Face
3. Try using a Hugging Face token with -t
4. Check if the model repository exists and is public
5. Try --remote mode for gated models

Error: Failed to build llama.cpp

Solutions:

1. Check if you have a C++ compiler installed
2. Ensure you have sufficient disk space
3. Try building with CPU-only mode if GPU builds fail
4. Update your GPU drivers if using acceleration

For special cases, you may want to modify llama.cpp's build parameters. The converter automatically detects and enables available hardware acceleration, but you can also build llama.cpp manually with custom options before running the converter.

The converter uses the latest llama.cpp features including:

• GGML_CUDA for NVIDIA GPU acceleration
• GGML_METAL for Apple Silicon
• GGML_VULKAN for cross-platform GPU support
• GGML_OPENCL for OpenCL acceleration
• GGML_HIPBLAS for AMD ROCm support