Generate speech, sound effects, music and beyond.
- Try to use more adjectives to describe your sound. For example: "A man is speaking clearly and slowly in a professional studio" is better than "A man is speaking". This can make sure AudioLDM understand what you want.
- Try to use different random seeds, which can affect the generation quality significantly sometimes.
- It's best to use general terms like 'man' or 'woman' instead of specific names for individuals or abstract objects that humans may not be familiar with, such as 'mummy'.
- Prepare running environment
conda create -n audioldm python=3.8; conda activate audioldm
pip3 install audioldm==0.0.6
git clone https://github.com/haoheliu/AudioLDM; cd AudioLDM
- Start the web application (powered by Gradio)
python3 app.py
- A link will be printed out. Click the link to open the browser and play.
- Prepare running environment
# Optional
conda create -n audioldm python=3.8; conda activate audioldm
# Install AudioLDM
pip3 install audioldm==0.0.6
- text-to-audio generation
# Test run
audioldm -t "A hammer is hitting a wooden surface"
For more options on guidance scale, batchsize, seed, etc, please run
audioldm -h
For the evaluation of audio generative model, please refer to audioldm_eval.
Integrated into Hugging Face Spaces ๐ค using Gradio. Try out the Web Demo
- Update the checkpoint with more training steps.
- Add AudioCaps finetuned AudioLDM-S model
- Build pip installable package for commandline use
- Build Gradio web application
- Add text-guided style transfer
- Add audio super-resolution
- Add audio inpainting
If you found this tool useful, please consider citing
@article{liu2023audioldm,
title={AudioLDM: Text-to-Audio Generation with Latent Diffusion Models},
author={Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D},
journal={arXiv preprint arXiv:2301.12503},
year={2023}
}
- GPU with 8GB of dedicated VRAM
- A system with a 64-bit operating system (Windows 7, 8.1 or 10, Ubuntu 16.04 or later, or macOS 10.13 or later) 16GB or more of system RAM
Part of the code is borrowed from the following repos. We would like to thank the authors of these repos for their contribution.
We build the model with data from AudioSet, Freesound and BBC Sound Effect library. We share this demo based on the UK copyright exception of data for academic research.