2021-11-06: I have just updated the code structure to make it easier to understand. It may have potential bug now. I will do some test training later.
2021-11-01: I will update the code and make it easier to use later.
VoiceFixer is a framework for general speech restoration. We aim at the restoration of severely degraded speech and historical speech.
- Arxiv preprint: https://arxiv.org/abs/2109.13731
- Demo page contains comparison between single task speech restoration, general speech restoration, and voicefixer.
- We wrote a pip package for voicefixer.
- The dataset we use in this repo: training and testing datasets
# Download dataset and prepare running environment
git clone https://github.com/haoheliu/voicefixer_main.git
cd voicefixer_main
source init.sh Here we take VF_UNet(voicefixer with unet as analysis module) as an example.
- Training
# pass in a configuration file to the training script
python3 train_gsr_voicefixer.py -c config/vctk_base_voicefixer_unet.json # you can modify the configuration file to personalize your trainingYou can checkout the logs directory for checkpoints, logging and validation results.
- Evaluation
Automatic evaluation and generating .csv file on all testsets.
For example, if you like to evaluate on all testset (default).
python3 eval_gsr_voicefixer.py \
--config <path-to-the-config-file> \
--ckpt <path-to-the-checkpoint> For example, if you just wanna evaluate on GSR testset.
python3 eval_gsr_voicefixer.py
--config <path-to-the-config-file> \
--ckpt <path-to-the-checkpoint> \
--testset general_speech_restoration \
--description general_speech_restoration_eval There are generally seven testsets you can pass to --testset:
- base: all testset
- clip: testset with speech that have clipping threshold of 0.1, 0.25, and 0.5
- reverb: testset with reverberate speech
- general_speech_restoration: testset with speech that contain all kinds of random distortions
- enhancement: testset with noisy speech
- speech_super_resolution: testset with low resolution speech that have sampling rate of 2kHz, 4kHz, 8kHz, 16kHz, and 24kHz.
And if you would like to evaluate on a small portion of data, e.g. 10 utterance. You can pass the number to --limit_numbers argument.
python3 eval_gsr_voicefixer.py \
--config <path-to-the-config-file> \
--ckpt <path-to-the-checkpoint> \
--limit_numbers 10 Evaluation results will be presented in the exp_results folder.
- Training
# pass in a configuration file to the training script
python3 train_gsr_voicefixer.py -c config/vctk_base_voicefixer_unet.jsonYou can checkout the logs directory for checkpoints, logging and validation results.
- Evaluation (similar to voicefixer evaluation)
python3 eval_ssr_unet.py --config <path-to-the-config-file> \ --ckpt <path-to-the-checkpoint> \ --limit_numbers <int-test-only-on-a-few-utterance> \ --testset <the-testset-you-want-to-use> \ --description <describe-this-test>
-
Training
- Denoising
# pass in a configuration file to the training script python3 train_ssr_unet.py -c config/vctk_base_ssr_unet_denoising.json- Dereverberation
# pass in a configuration file to the training script python3 train_ssr_unet.py -c config/vctk_base_ssr_unet_dereverberation.json- Super Resolution
# pass in a configuration file to the training script python3 train_ssr_unet.py -c config/vctk_base_ssr_unet_super_resolution.json- Declipping
# pass in a configuration file to the training script python3 train_ssr_unet.py -c config/vctk_base_ssr_unet_declipping.json
You can checkout the logs directory for checkpoints, logging and validation results.
- Evaluation (similar to voicefixer evaluation)
python3 eval_ssr_unet.py --config <path-to-the-config-file> \ --ckpt <path-to-the-checkpoint> \ --limit_numbers <int-test-only-on-a-few-utterance> \ --testset <the-testset-you-want-to-use> \ --description <describe-this-test>
@misc{liu2021voicefixer,
title={VoiceFixer: Toward General Speech Restoration With Neural Vocoder},
author={Haohe Liu and Qiuqiang Kong and Qiao Tian and Yan Zhao and DeLiang Wang and Chuanzeng Huang and Yuxuan Wang},
year={2021},
eprint={2109.13731},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent