xcmyz / large-audio-models

Keep track of big models in audio domain, including speech, singing, music etc.

Large-Audio-Models

We keep track of something big in the audio domain, including speech, singing, music etc.

Contents

• Prompt-based Audio Synthesis
• Audio Language Models
• Audio SSL/UL models

Prompt-based Audio Synthesis

• VALL-E X: Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023), Ziqiang Zhang et al. [PDF]

• Noise2Music: Text-conditioned Music Generation with Diffusion Models(2023), Qingqing Huang et al. [PDF]

• MusicLM: Generating Music From Text(2023), Andrea Agostinelli et al. [PDF]

• InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt (2023), Dongchao Yang et al. [[PDF]] (https://arxiv.org/pdf/2301.13662.pdf)

• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models(2023), Haohe Liu et al. [PDF]

• Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion(2023), Flavio Schneider et al. [PDF]

• VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023), Chengyi Wang et al. [PDF]

Audio Language Models

• AudioLM: a Language Modeling Approach to Audio Generation(2022), Zalán Borsos et al. [PDF]

Audio SSL and UL models

• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021) Wei-Ning Hsu et al. [PDF],
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020), Alexei Baevski et al. [PDF]