LoRA Reference

LoRA is a parameter-efficient fine-tuning technique that injects trainable low-rank matrices into pre-trained weights, typically around linear layers. Compared with full-parameter fine-tuning, this reduces memory usage and compute cost substantially, making RL fine-tuning of large models much more practical on limited hardware.

In AReaL, this is especially useful for:

• reinforcement learning with very large models, including 70B+ models, on relatively modest hardware such as 8 x 80 GB GPUs,
• enabling larger batch sizes because LoRA reduces training memory pressure,
• simplifying transfer and deployment because only the LoRA adapters need to be saved and shipped,
• [Future] fine-tune multiple LoRA adapters more efficiently in parallel for better hardware utilization (see RFC #609).

This guide explains how to enable LoRA in RL training and configure the related parameters.

Backend Support

The current LoRA support matrix in AReaL is:

Engine	vLLM	SGLang
FSDP2	✅	✅
Megatron	✅	❌
Archon	❌	❌

Example scripts:

Engine	Example script
FSDP2	examples/math/gsm8k_grpo_lora.yaml
Megatron	examples/math/gsm8k_grpo_megatron_lora.yaml
Megatron MoE	examples/math/gsm8k_grpo_megatron_lora_moe.yaml

For Megatron + vLLM, AReaL now supports:

• LoRA fine-tuning on MoE architectures such as Qwen3 MoE with XCCL-based LoRA weight.
• Cross-node LoRA training when the Megatron and rollout groups span multiple nodes.

Core LoRA Parameters

Parameter	What it controls	Typical values
use_lora	Enables LoRA fine-tuning mode.	true / false
lora_rank (r)	Rank of the low-rank adapters. Higher rank increases capacity and memory/compute cost.	8, 16, 32, 64
lora_alpha	LoRA scaling factor. Effective adapter scale is commonly thought of as proportional to alpha / r.	16, 32, 64
target_modules	Which model submodules receive LoRA adapters. This is the most important architecture-specific setting.	e.g. [all-linear]
peft_type	PEFT method type. In AReaL configs, this is LoRA.	lora

Practical Notes

• Start with r=16 or r=32 for most models, then tune upward only if needed.
• Keep target_modules consistent with your model architecture naming.
• For Megatron backend, LoRA requires megatron-bridge instead of mbridge.