For spanish preprocessing text:
python preprocess.py --text_index 1 --text_cleaners spanish_cleaners --filelists /path/to/train/filelist.txt /path/to/val/filelist.txt
If you using and ocmpiling the most recent version of espeak-ng, you can preprocess the text in Latin American Spanish (if you want to train a voice with that dialect) using the following:
python preprocess.py --text_index 1 --text_cleaners spanish_cleaners2 --filelists /path/to/train/filelist.txt /path/to/val/filelist.txt
if you get this warning when preprocessing, ignore it.
WARNING:phonemizer:words count mismatch on 100.0% of the lines (1/1)
WARNING:phonemizer:words count mismatch on 100.0% of the lines (1/1)
WARNING:phonemizer:words count mismatch on 100.0% of the lines (1/1)
For training, remember to update the configs/base_es_singlespeaker_22k.json
or configs/base_es-latin_singlespeaker_22k
file with the paths of your training and validation files, as well as model settings such as batch size and number of epochs to train. This depends of your dataset.
"name": "your_speaker_name",
"training_files": "/path/to/train/filelist.txt",
"validation_files": "/path/to/val/filelist.txt",
TRAINING IT´S NOW WORKING!
Start training with
python train.py -c configs/base_es_singlespeaker_22k.json -m "voice_name" -p "output_path/to/save"
- Try to implement LoRA Finetuning on VITS by modifying attentions.py as described in the LoRA Paper
- Doesnt work as the generator doesnt get updated, need to research more
- Try to implement 8-bit training using bitsandbytes to reduce vram usage
- 8 bit optimisers
- Write paper explaining steps?
- Extract discriminator from hifigan
- Modify models.py to match hifigan implementation
- Make sure training loop runs
- Clean up logging, maybe integrate WandB to track training progress
- WandB integrated! Run
train_wandb.py
to use it
- WandB integrated! Run
- Test finetune as-is on test dataset to make sure patched discriminator works
- Wont be releasing weights for the optimusPrime as model is not good. See w&b report underneath
- Will be trying to train with a better dataset to showcase the finetune
- implement LoRA
- Test finetune on lora using same test dataset
- Implement 8-bit optimizers
- Create webpage to show results (see w&b report)
- The generator for LJSpeech is here
- The discriminator(extracted from hifi-gan) is here
- A notebook with a running training loop is here
- W&B Report on finetuning with 8bit optimisers
- I realise that if this works, anyone remotely knowledgeable about machine learning will be able to finetune VITS on large datasets quite quickly to achieve pretty good voice cloning. However, this is already a thing(tortoise-tts: mrq fork, DLAS fork), but it takes a long time to finetune and generate.
- However, sparks of efficient finetuning for TTS systems are already here, its only a matter of time before someone like me will do it for other models.
- My initial plan is to only provide a comparison of results, and any good and ethical finetunes. I believe the real value lies in the dataset creation so will not be opensourcing any scripts to clean and curate the data.
In our recent paper, we propose VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech.
Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.
Visit our demo for audio samples.
We also provide the pretrained models.
** Update note: Thanks to Rishikesh (ऋषिकेश), our interactive TTS demo is now available on Colab Notebook.
VITS at training | VITS at inference |
---|---|
![]() |
![]() |
- Python >= 3.6
- Clone this repository
- Install python requirements. Please refer requirements.txt
- You may need to install espeak first:
apt-get install espeak-ng
- You may need to install espeak first:
- Download datasets
- Download and extract the LJ Speech dataset, then rename or create a link to the dataset folder:
ln -s /path/to/LJSpeech-1.1/wavs DUMMY1
- For mult-speaker setting, download and extract the VCTK dataset, and downsample wav files to 22050 Hz. Then rename or create a link to the dataset folder:
ln -s /path/to/VCTK-Corpus/downsampled_wavs DUMMY2
- Download and extract the LJ Speech dataset, then rename or create a link to the dataset folder:
- Build Monotonic Alignment Search and run preprocessing if you use your own datasets.
# Cython-version Monotonoic Alignment Search
cd monotonic_align
python setup.py build_ext --inplace
# Preprocessing (g2p) for your own datasets. Preprocessed phonemes for LJ Speech and VCTK have been already provided.
# python preprocess.py --text_index 1 --filelists filelists/ljs_audio_text_train_filelist.txt filelists/ljs_audio_text_val_filelist.txt filelists/ljs_audio_text_test_filelist.txt
# python preprocess.py --text_index 2 --filelists filelists/vctk_audio_sid_text_train_filelist.txt filelists/vctk_audio_sid_text_val_filelist.txt filelists/vctk_audio_sid_text_test_filelist.txt
# LJ Speech
python train.py -c configs/ljs_base.json -m ljs_base
# VCTK
python train_ms.py -c configs/vctk_base.json -m vctk_base
See inference.ipynb