Will this work for singing voice conversion (svc)? about knn-vc

[RF5]

Hi @billnye2 , thanks for your comments :). Some thoughts:

• Yep we also found kNN-VC to not do super well with singing, especially for more expressive / melodic songs.
• It is largely expected, since the two trained parts of kNN-VC (the WavLM encoder and HiFiGAN vocoder) are both only trained on English librispeech, which is fairly monotone. Both of these hurt quality when presented with singing inputs. The kNN part is quite agnostic to singing vs non-singing vs non-human sounds, so likely the main limitation is from the feature encoding and vocoding side.
• To fix the WavLM side, I think one would need to retrain WavLM with some singing data added, so that the features it produces can better represent singing audios.
• To fix the HiFiGAN vocoding side, using NSF-HiFiGAN might definitely improve things. I imagine what would be required would be to train an NSF-HiFiGAN model to vocode WavLM features (instead of spectrograms), then it can be directly used with kNN-VC.

Hopefully in the not too distant future, we will be able to generalize the performance of kNN-VC and other models. Thank's again for your interest in our work!

[billnye2]

Great support, thank you!