Instruction Manual

February 18, 2025 ยท View on GitHub

This is the implementation of the paper "Emotion and Intensity Regulated Text-to-Speech Synthesis Through Prompt Guidance"

Credited to Comprehensive-TTS and Relative Attribute

Sample Audios

Some samples generated by our framework are located at Audio_Samples folder. It contains the samples used in Mean Opinion Score(MOS) test and Perceptual Intensity Ranking (test). MOS folder contains the audios generated by three different models, while the PIR folder contains the audios generated by our proposed model(FastSpeech 2 with Emotion and Intensity Encoder)

Figure visulization is located at sources/figures

Environment

Make sure the workstation has conda installed before setting up the environment

conda update conda
conda create -n proemo python=3.8
conda activate proemo
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 -c pytorch
pip install -r requirements.txt

The PyTorch installation will be case-by-case, depending on the CUDA version and operating system. Refer to the official website for the compatible installation.

Resources Download

Download LibriTTS, LJSpeech(optional), ESD, IMOCap(optional), then allocate them in Audio_Data/datasets

Download openSMILE toolkit for Learned Rank Function feature extraction.

Download related data of this repo here

Learned Ranked Function(LRF)

Derive intensity with LRF algorithm

python learned_rank_func/preprocess_os.py --dataset DATASET
python learned_rank_func/training.py --dataset DATASET

The entire process could take 1 hour for either ESD or IMOCap. Alternatively, use normalized intensity scores under Audio_Data/preprocessed_data/DATASET/relative_attr

Dataset Preprocess

The workflow of preprocessing is taken from Comprehensive-TTS. The adapted scripts preprocess Pitch/Energy/Duration in phoneme-level, we don't use supervised duration modelling as well. DATASET is one of LibriTTS, ESD. The train|test split could be found in Audio_Data/preprocessed_data/DATASET

python prepare_align.py --dataset DATASET
python preprocess.py --dataset DATASET

Training

### Pretrain
python train.py --dataset LibriTTS 
### Fine-Tune 
python train.py --dataset ESD --restore_step 900000 --checkpoint LibriTTS --emotion_label 1 --intensity_label 1

For emotional datasets, emotion_label should be 1, intensity_label can be 0 or 1. The training could take days with single GPU setting. If you download the related data, you can direcly use the trained weights at Audio_Data/output/CHECKPOINT/ckpt/RESTORED_STEP.pth.tar

Monitor the training

You can monitor the training process by running the following command:

tensorboard --logdir Audio_Data/output/CHECKPOINT --port 4000 --host 0.0.0.0
# or
tensorboard --logdir Audio_Data/output/CHECKPOINT --port 4000 --bind_all

port number could vary based on your needs. A link will be popped up to access the tensorboard.

Sythesize audios

Sythesize the embeddings (emotion or intensity) first.

python generate_embeds.py --checkpoint ESD --restore_step 450000 --label_type emotion
# and 
python generate_embeds.py --checkpoint ESD --restore_step 450000 --label_type intensity

There is no difference if restore_step is set as 50000, as the maximum encoder training step is 50000. You can execute the objective evaluation and subjective evalution to obtain sythesized audios accross different settings. The objective evaluation is reconstructed from the validation set, while the subjective evaluation is reconstructed from any text. You can run the following command:

python objective_eval.py --source Audio_Data/preprocessed_data/ESD_unsup/val_prompt_unsup.txt --restore_step 450000 --checkpoint Audio_Data/output/CHECKPOINT --dataset DATASET --emotion_label 1 --intensity_label 0
python objective_eval.py --source Audio_Data/preprocessed_data/ESD_unsup/val_prompt_unsup.txt --restore_step 450000 --checkpoint Audio_Data/output/CHECKPOINT --dataset DATASET --emotion_label 1 --intensity_label 1

The scripts are designed for ESD and IMOCap dataset, but IMOCap checkpoints are not available yet due to performance issue. You may also run the subjective evaluation by running the following command:

python subjective_eval.py --source val_mos.txt --restore_step 450000 --mode batch --checkpoint ESD --label emotion

val_mos.txt, val_bws.txt is free-form any input text, while val_prompt_unsup.txt is from the validation set. The checkpoint is the name of the folder in Audio_Data/output.

# python subjective_eval.py --source val_bws.txt --restore_step 450000 --mode batch --checkpoint ESD --label intensity
python subjective_eval.py --source Audio_Data/preprocessed_data/ESD_unsup/val_prompt_unsup.txt --restore_step 450000 --mode batch --checkpoint ESD --label intensity

TODO Notes 2025

  1. generate_embeds.py: Test scripts that only loads the encoder models, instead of the entire TTS model.
  2. evaluation: clarify the explanation
  3. Add credits and collaborators
  4. Add the plot functions.