Missing words in audio, improper pauses about transformertts

[cfrancesco]

I would like to use the code to generate some audio samples as voice-overs for some tutorials I'm creating. I've been testing the code out, and as @aayushkubb mentioned, there are missing words and other odd behaviour. I do understand though that there are some technical limitations mentioned.

My machine learning background is fairly lacking so excuse my terminology and lack of understanding, however, I would like to use the 'Autoregressive + MelGAN' colab since it sounds best. Would it be possible to use the Autoregressive model to only train the forward model (MelGAN), and then feed a custom sentence to MelGAN for it vocalize?

In terms of code, that doesn't seem so possible. The relevant bits after dissecting it a bit are the sections below. I would like to feed the torch.tensor() function a custom sentence like the Autoregressive portion and not the output audio file of the Autoregressive. Does this even make sense in the context of this machine learning algorithm?

# Synthesize text
sentence = 'Scientists at the CERN laboratory, say that they have discovered a new particle.'
out = model.predict(sentence)

sys.path.append(MelGAN_path)
import torch
import numpy as np

vocoder = torch.hub.load('seungwonpark/melgan', 'melgan')
vocoder.eval()

mel = torch.tensor(out['mel'].numpy().T[np.newaxis,:,:])

I'm not sure about the last portion of you message, but I would recommend you use the Forward model for TTS (for highest stavbility, autoregressive is just very unstable) and convert with WaverRNN for the highest quality.

The thing you feed to the vocoder (either MelGAN or WaverRNN) are spectrogram predictions from TTS models, so neither text nor audio.

If you have more questions please open a new issue.

[cfrancesco]

Yes, the autoregressive model is unstable and hard to train. It should only be used to train the forward model.
Pauses are hard to predict if there is a lot of variations and skipping words is a known issue.

Are the decoder heads aligning properly? (check this from tensorboard)

[othmanalikhan]

I would like to use the code to generate some audio samples as voice-overs for some tutorials I'm creating. I've been testing the code out, and as @aayushkubb mentioned, there are missing words and other odd behaviour. I do understand though that there are some technical limitations mentioned.

My machine learning background is fairly lacking so excuse my terminology and lack of understanding, however, I would like to use the 'Autoregressive + MelGAN' colab since it sounds best. Would it be possible to use the Autoregressive model to only train the forward model (MelGAN), and then feed a custom sentence to MelGAN for it vocalize?

In terms of code, that doesn't seem so possible. The relevant bits after dissecting it a bit are the sections below. I would like to feed the torch.tensor() function a custom sentence like the Autoregressive portion and not the output audio file of the Autoregressive. Does this even make sense in the context of this machine learning algorithm?

# Synthesize text
sentence = 'Scientists at the CERN laboratory, say that they have discovered a new particle.'
out = model.predict(sentence)

sys.path.append(MelGAN_path)
import torch
import numpy as np

vocoder = torch.hub.load('seungwonpark/melgan', 'melgan')
vocoder.eval()

mel = torch.tensor(out['mel'].numpy().T[np.newaxis,:,:])