GigaAM Fine-tuning (CTC / RNNT)

For training and evaluation examples and a more detailed description of manifests, see example.ipynb.

Setup

From the repository root, install the training dependencies:

pip install -e ".[train]"

cd train_utils

Data format

TSV manifest (tab-separated columns): path, duration, and optionally transcription. Paths may be absolute or relative to the manifest directory. transcription may be omitted for audio-only manifests.

path	duration	transcription
audio/0001.wav	3.21	привет как дела

Training

python train.py \
    --model_name v3_e2e_ctc \
    --train_manifest /path/to/manifest_train.tsv \
    --val_manifest /path/to/manifest_val.tsv \
    --max_epochs 3 \
    --val_check_interval 0.5 \
    --batch_size 64 \
    --eval_batch_size 64 \
    --lr 8e-5 \
    --activation_checkpointing

python train.py \
    --model_name v3_rnnt \
    --train_manifest /path/to/manifest_train.tsv \
    --val_manifest /path/to/manifest_val.tsv \
    --raw_text \
    --max_epochs 2 \
    --val_check_interval 0.5 \
    --batch_size 8 \
    --accumulate_grad_batches 2 \
    --rnnt_subbatch_size 2 \
    --eval_batch_size 64 \
    --lr 2e-5 \
    --val_first_batches 50

Fine-tuning a new language from an SSL backbone

The *_ssl models (e.g. multilingual_ssl) are self-supervised backbones: a preprocessor and a Conformer encoder, without an ASR decoder. To adapt one to a new language (Georgian, Bashkir, ...), pass the SSL model as --model_name together with a target vocabulary — train.py then attaches a freshly initialized character-wise head and fine-tunes it. Choose the head with --head ctc (default) or --head rnnt.

The vocabulary is either derived from your training transcriptions or supplied as a file. With --raw_text (recommended) the labels are normalized charwise (lowercase, punctuation stripped) and the derived vocabulary follows; without it the verbatim transcriptions are used (e2e-style, case and punctuation become output classes). A file-supplied vocabulary is used as-is — mismatches against the manifests are reported as warnings:

# CTC head, alphabet derived straight from the manifest transcriptions
python train.py \
    --model_name multilingual_ssl \
    --head ctc \
    --train_manifest /path/to/ka_train.tsv \
    --val_manifest /path/to/ka_val.tsv \
    --build_vocab_from_manifest \
    --save_vocab ./ka_vocab.json \
    --raw_text \
    --max_epochs 5 \
    --val_check_interval 0.5 \
    --batch_size 16 \
    --lr 1e-4 \
    --activation_checkpointing

# RNN-T head, with an explicit vocabulary file
python train.py \
    --model_name multilingual_ssl \
    --head rnnt \
    --train_manifest /path/to/ba_train.tsv \
    --val_manifest /path/to/ba_val.tsv \
    --vocab ./ka_vocab.json \
    --raw_text \
    --rnnt_subbatch_size 2 \
    --max_epochs 5 --batch_size 16 --lr 1e-4

Arguments

Model and data

Argument	Default	Description
--model_name	required	Pretrained GigaAM model name
--train_manifest	required	TSV manifest for training
--val_manifest	required	TSV manifest for validation
--head	ctc	Head to attach when fine-tuning from an SSL backbone: ctc or rnnt (ignored for trained ASR models)
--rnnt_pred_hidden	320	RNN-T predictor hidden size (SSL --head rnnt)
--rnnt_pred_rnn_layers	1	RNN-T predictor LSTM layers (SSL --head rnnt)
--rnnt_joint_hidden	320	RNN-T joint network hidden size (SSL --head rnnt)
--raw_text	off	For non-E2E setups: lowercase text, drop punctuation, restrict to the character vocabulary
--max_duration	20.0	Maximum audio length in seconds (dataset filter)
--min_duration	0.1	Minimum audio length in seconds (dataset filter)
--vocab	None	Target vocabulary file (SSL backbone only): a .json list
--build_vocab_from_manifest	off	Derive the character vocabulary from the train manifest transcriptions (SSL backbone only)
--save_vocab	None	Optional path to dump the resolved vocabulary as JSON

Scheduling (epochs vs steps)

Argument	Default	Description
--max_epochs	None	Train for this many epochs (omit if using --max_steps)
--val_check_interval	1.0	Run validation every N-th fraction of an epoch
--max_steps	None	Train for this many steps (requires --val_check_steps)
--val_check_steps	None	With --max_steps: validate every N training steps
--val_first_batches	None	If set, run validation on only the first N batches

Batching and memory

Argument	Default	Description
--batch_size	8	Per-device training batch size
--eval_batch_size	32	Validation batch size
--rnnt_subbatch_size	0	RNNT loss sub-batches (0 disables)
--num_workers	4	Number of DataLoader workers
--precision	32	Lightning precision (16, bf16, 32, ...)
--accumulate_grad_batches	1	Gradient accumulation steps
--accelerator	auto	Lightning accelerator (auto, cpu, gpu, ...)
--devices	1	Number of devices (DDP when > 1)
--activation_checkpointing	off	Activation checkpointing for each Conformer layer
--freeze_encoder_epochs	0	-1 — freeze encoder weights for the entire run; 0 — usual fine-tuning (encoder always trainable); N > 0 — staged fine-tuning: encoder is frozen for the first N epochs, then unfrozen

Optimizer

Argument	Default	Description
--lr	2e-5	Peak learning rate (AdamW)
--weight_decay	0.01	AdamW weight decay
--warmup_ratio	0.1	Linear warmup fraction before cosine decay
--gradient_clip_val	1.0	Global gradient norm clipping
--seed	42	Passed to pl.seed_everything

SpecAugment

Argument	Default	Description
--freq_masks	2	Number of frequency masks
--freq_width	27	Maximum width of each frequency mask (bins)
--time_masks	2	Number of time masks
--time_width	20	Maximum width of each time mask (frames)
--disable_spec_augment	off	Disable SpecAugment (enabled by default)

Outputs, logging and resuming

Argument	Default	Description
--output_dir	./checkpoints	Checkpoints under models/<exp_name>/, TensorBoard under tb_logs/
--exp_name	auto	Run directory name; if omitted, derived from hyperparameters
--log_every_n_steps	25	Logging interval in steps
--save_top_k	2	Keep this many best checkpoints by val_wer
--disable_tqdm	off	Disable progress bars
--skip_initial_validation	off	Skip trainer.validate before fit
--resume_from_checkpoint	None	Path to a Lightning .ckpt file to resume training from

Evaluation

Evaluate a fine-tuned checkpoint:

python eval.py \
    --checkpoint ./checkpoints/models/<exp_name>/gigaam-*.ckpt \
    --eval_manifest /path/to/manifest.tsv

Evaluate a pretrained GigaAM model:

python eval.py --model_name v3_e2e_ctc --eval_manifest /path/to/manifest.tsv

This writes preds.jsonl and prints WER. Predictions are saved under predictions/<manifest_stem>/<exp_name>/step_<step>/preds.jsonl (step_<step> is omitted for pretrained models) next to the manifest. WER is reported on the original transcripts (end-to-end) and on raw texts.

Loading fine-tuned checkpoints

gigaam.load_model accepts a path to a Lightning .ckpt file, so fine-tuned models can be loaded the same way as pretrained ones:

model = gigaam.load_model("./checkpoints/models/<exp_name>/gigaam-*.ckpt")