Soup

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Website · Quick Start · Config · Docs · Commands · Models · Discord

Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.

pip install "soup-cli[train]"   # add [train] to fine-tune; bare `soup-cli` is the light CLI
soup init --template chat
soup train

Fine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop 4 GB: Llama-3.1-8B-Instruct + NF4 at 119.6 tok/s, 3.32 GB peak — bit-exact against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB. (The tok/s figure was measured on v0.72.2, before the v0.73.0 correctness repair that cost −4.8% at 32B; it has not been re-run on a 4 GB card since.) Opt-in (stream_layers: true) and still BETA — how it works · all measurements · paper

_{Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — 3.32 GB peak, 119.6 tok/s. Full video (90s)}

Why Soup?

Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.

• Zero SSH. Never SSH into a broken GPU box again.
• One config. A simple YAML file is all you need.
• Auto everything. Batch size, GPU detection, quantization — handled.
• Works locally. Train on your own GPU with QLoRA. No cloud required.

What's New

v0.73.0 — three days on somebody else's hardware. Every number Soup had ever published came from one machine: a 4 GB RTX 3050 laptop running Windows. From 5–9 August it ran on a borrowed 8×H100 box. That found real bugs, and it confirmed the headline claim on hardware nothing like the one it was made on.

• The laptop result reproduces elsewhere. Llama-3.1-8B NF4 streamed: 119.6 tok/s in a 3.32 GB peak on the RTX 3050, against a median 113.00 tok/s in the same 3.32 GB on an H100. Layer streaming is bound by host-to-device transfer, not by the GPU.
• A silent wrong-gradient bug, found and fixed. On NF4 models above ~165 MiB per layer (32B and up), bitsandbytes kept a weight reference where gradient checkpointing could not see it, so the forward stayed exact and the loss curve looked healthy while the gradients were wrong. Repaired and gated on real 32B (256/256 gradient tensors exact against a control's 8–12/256) and real 72B (320/320 against 8/320), at −4.8% and −3.7% throughput.
• Four backends that had never actually run, now do. soup train --gpus N handed accelerate the Python binary and every rank died parsing it as source. DeepSpeed could not train a LoRA model on any stage. SGLang returned 500 on 100% of generations. And use_fsdp2_compile wrote adapters that reload as all zeros (0 of 96 tensors live).
• The vLLM backend now uses your model's chat template, instead of a hand-rolled "User:/Assistant:" string it was never trained on. Same server, same sampling: a run-on loop burning 200 tokens before, an 8-token answer after.
• New: training.seed (every run was hardcoded to 42) and full fine-tuning via lora.r: 0 (the code path existed but was unreachable).
• A streamed model is as good as a resident one — paired over five training subsets and judged by Soup's own soup ship: mean difference +0.006 against a 0.013 within-arm spread.

The full measurement record, published as written including the rejected hypotheses and the false positives that controls caught, is benchmarks/gate-h100-validation.md.

# soup.yaml — then just `soup train --config soup.yaml`
training:
  stream_layers: true      # base streams out of VRAM; only the adapter trains
  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
  batch_size: 4            # bigger batches amortise the weight read
  stream_source: auto      # RAM when it fits, NVMe disk when it does not
  seed: 1234               # new in v0.73.0

Python 3.10–3.12 only. v0.73.0 adds the upper bound that was missing: on 3.13+, pip used to resolve untested PyTorch wheels that crash in the native extension before Soup runs at all.

soup reward synth references.jsonl -o reward.py --output-report calib.json

soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error

Full history: CHANGELOG.md · GitHub Releases.

Quick Start

1. Install

# Light core: CLI + config + data tools, no PyTorch
pip install soup-cli

# Add the training stack (torch, transformers, peft, trl, datasets, …)
pip install "soup-cli[train]"

# Everything (train + serve + ui + data) in one shot
pip install "soup-cli[all]"

# Or from GitHub (latest dev)
pip install git+https://github.com/MakazhanAlpamys/Soup.git

The full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in docs/models.md.

Use double quotes around the extra. They are the only spelling that works in every shell — cmd.exe, PowerShell, bash, and zsh.

Older tutorials and videos (including some of ours) show the single-quoted pip install 'soup-cli[train]'. That is bash / zsh / PowerShell syntax, and it fails on Windows cmd.exe, which has no single-quote quoting and hands the quotes straight to pip:

ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier

If you hit that, swap the ' for " — pip is rejecting a literal quote character, nothing is wrong with the package. (Dropping the quotes entirely works on Windows too, but zsh then reads [train] as a glob and fails.)

soup init, soup data …, and the other data/inspection commands work on the light install. Fine-tuning (soup train) needs the [train] extra.

2. Create a config

soup init                       # interactive wizard
soup init --template chat       # or start from a template

Templates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo, simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.

3. Train, test, ship

soup train --config soup.yaml                 # LoRA, quantization, batching — all handled
soup chat  --model ./output                    # talk to your model
soup push  --model ./output --repo you/my-model

soup merge  --adapter ./output                              # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m   # GGUF for Ollama / llama.cpp

More export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in docs/serving-and-export.md.

Configuration

A complete soup.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth  # 2-5x faster, pip install "soup-cli[fast]"

data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

output: ./output

config/schema.py is the single source of truth for every field. Advanced data, training, and PEFT options are documented under Documentation.

Documentation

The full feature reference lives in docs/. Start here:

Data Formats

All formats are auto-detected from JSONL, JSON, CSV, Parquet, or TXT:

• alpaca — {"instruction": ..., "input": ..., "output": ...}
• sharegpt — {"conversations": [{"from": "human", "value": ...}, ...]}
• chatml — {"messages": [{"role": "user", "content": ...}, ...]}
• dpo / orpo / simpo / ipo — {"prompt": ..., "chosen": ..., "rejected": ...}
• kto — {"prompt": ..., "completion": ..., "label": true}
• llava / sharegpt4v (vision), audio, plaintext (pre-training), embedding, prm, pre_tokenized, video, multimodal

Full schemas and the Axolotl/LlamaFactory-parity data pipeline (remote URIs, streaming, sharding, interleaving, vocab expansion, document ingestion) are in docs/data.md.

Common Commands

soup train  --config soup.yaml        # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)
soup infer  --model ./output --input prompts.jsonl   # batch inference
soup chat   --model ./output          # interactive chat
soup serve  --model ./output          # OpenAI-compatible API server
soup merge  --adapter ./output        # merge LoRA into the base model
soup export --model ./output --format gguf           # export for deployment
soup eval   benchmark --model ./output               # evaluate
soup data   inspect ./data/train.jsonl               # dataset stats
soup recipes list                     # 100+ ready-made model recipes
soup autopilot --model <id> --data d.jsonl --goal chat  # zero-config
soup doctor                           # check GPU / deps / environment

The complete command list is in docs/commands.md.

Supported Models

Soup works with any text-generation model on the HuggingFace Hub — if it loads with AutoModelForCausalLM, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral, Mixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (soup recipes list).

Full model + vision tables and the optional-extras matrix are in docs/models.md.

Docker

Run Soup without installing CUDA or PyTorch locally (image published to GHCR on every release):

docker pull ghcr.io/makazhanalpamys/soup:latest
docker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml
docker compose up   # or build locally

Requirements

• Python 3.10, 3.11 or 3.12 (those are the versions CI tests; 3.13+ is not supported yet because the PyTorch stack has not been validated there)
• GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)
• 8 GB+ VRAM for 7B models with QLoRA

All training tasks run on CPU for testing (quantization auto-disabled). Optional extras (train, all, fast, vision, qat, serve, serve-fast, ui, eval, deepspeed, liger, mlx, onnx, tensorrt, …) are listed in docs/models.md.

Troubleshooting

soup doctor    # GPU, system resources, dependencies, and version in one place

• ImportError: DLL load failed while importing _C (Windows) — reinstall PyTorch for your CUDA version: pip install torch --index-url https://download.pytorch.org/whl/cu121.
• soup version ≠ pip show soup-cli — multiple Python installs; use a virtualenv.

Development

git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
pip install -e ".[dev]"

ruff check src/soup_cli/ tests/    # lint
pytest tests/ -v                   # unit tests (fast, no GPU)
pytest tests/ -m smoke -v          # smoke tests (downloads a tiny model, trains)

pre-commit install                 # optional: ruff lint+format on commit

See CONTRIBUTING.md for the full workflow and SECURITY.md to report a vulnerability.

Support Soup

Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a single 4 GB laptop, which is why every performance number in these docs is measured rather than claimed.

If Soup saved you a training run, starring the repo helps most, and it costs nothing. If you would like to fund the work directly:

❤️ Donate — one-off, any amount (use Change amount on the checkout page). Payments are processed by Stripe under the maintainer's registered business, MePlay, Inc. — that name, not "Soup", is what appears on the checkout page and on your card statement.

Donations buy GPU time for the hardware-gated work — multi-GPU, 8B+ validation, Apple Silicon — that a single 4 GB laptop cannot reach.

The other way to move exactly those items is hardware itself. They ship behind honest "requires <hardware>" gates rather than unverified claims, so if you have access to a bigger box — or GPU credits going unused — running one of the help wanted issues and posting the numbers helps as much as funding the GPU time would. Those issues say exactly what is blocked on hardware today.

Contributors

Built by the community ❤️ — thank you to everyone who has contributed. See CONTRIBUTORS.md.

Contact

Bugs and feature requests belong in the issue tracker, questions in Discussions — both get answered faster and help the next person with the same problem.

For live chat, setup help, and everything that reads better as a conversation, join the Discord. Anything that should still be findable in six months belongs in Issues or Discussions — a Discord answer helps one person, an issue helps everyone who hits the same thing. The Code of Conduct applies there too.

For anything that does not fit in public — security reports (see SECURITY.md), Code of Conduct matters, or press — email team@trysoup.dev. That is the project address and the right one for anything Soup-related. makazanalpamys@gmail.com is the maintainer's personal address; it reaches the same person and is a fine fallback.

Citing Soup

Layer streaming — training an 8B model on a 4 GB laptop GPU by streaming the frozen base from host RAM one decoder layer at a time — is described in a preprint, together with the correctness protocol that verifies a streamed run is bit-exact against a resident one:

Makazhan, A. (2026). Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU. Zenodo. https://doi.org/10.5281/zenodo.21771064

The measurement records behind every number in it are in benchmarks/, published as written — including the failures, the assumptions that turned out wrong, and the numbers that were measured and then discarded.

@misc{makazhan2026exact,
  title        = {Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU},
  author       = {Makazhan, Alpamys},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21771064},
  url          = {https://doi.org/10.5281/zenodo.21771064}
}

License

Apache-2.0. Copyright © the Soup contributors.