Comments (2)
Hi @billnye2 , thanks for your comments :). Some thoughts:
- Yep we also found kNN-VC to not do super well with singing, especially for more expressive / melodic songs.
- It is largely expected, since the two trained parts of kNN-VC (the WavLM encoder and HiFiGAN vocoder) are both only trained on English librispeech, which is fairly monotone. Both of these hurt quality when presented with singing inputs. The kNN part is quite agnostic to singing vs non-singing vs non-human sounds, so likely the main limitation is from the feature encoding and vocoding side.
- To fix the WavLM side, I think one would need to retrain WavLM with some singing data added, so that the features it produces can better represent singing audios.
- To fix the HiFiGAN vocoding side, using NSF-HiFiGAN might definitely improve things. I imagine what would be required would be to train an NSF-HiFiGAN model to vocode WavLM features (instead of spectrograms), then it can be directly used with kNN-VC.
Hopefully in the not too distant future, we will be able to generalize the performance of kNN-VC and other models. Thank's again for your interest in our work!
from knn-vc.
Great support, thank you!
from knn-vc.
Related Issues (20)
- prematch argument HOT 2
- Choice for k HOT 2
- Conversion output has very strong similarity to source audio. HOT 4
- Considering context around source features HOT 2
- extend to other SSL model features
- Will this work with 44100hz audio? HOT 4
- Error in quickstart HOT 2
- Size mismatch error HOT 2
- loss issues encountered in fine-tuning the model HOT 6
- Maybe mention memory consumption in readme.md? HOT 3
- Using another batch size in training HOT 2
- Some questions about implementation HOT 2
- Matching pool empty HOT 2
- Hints on improvements for training and matching HOT 3
- How to plug-in new finetuned HiFiGAN? HOT 1
- Some question about features KNN HOT 2
- Request HOT 2
- Question about the Used Hardware HOT 1
- Using better neural vocoder GAN (BIGVGAN, RVQGAN ...) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from knn-vc.