CUDA Out-Of-Memory Error about multilingual_text_to_speech

[sooftware]

I've got a temporary system that keeps training automatically to the shell.

for (( ; ; ))
do
    for entry in "./checkpoints"/*
    do
        last_checkpoint=$entry
    done

    tokens=$(echo $last_checkpoint | tr "/" "\n")
    for tokenin $tokens
    do
       checkpoint="$token"
    done
    echo "Load checkpoint : $checkpoint"

    PYTHONIOENCODING=utf-8 python3 train.py --hyper_parameters generated_training --checkpoint $checkpoint
done

[Tomiinek]

This is really strange. I used 2 x 8GB GTX1080 for training and I have never experienced this problem 😕

The script you posted seems to be ok, I am not sure whether I can help you ... Couldn't be all the four runs executed on the same single GPU (i.e., the other three are left untouched)?

[sooftware]

Hmm. that`s weird. 😢😢
If I find a problem, I'll write it down here. Thanks for the answer.

[sooftware]

I trained including Korean and English.
So, as mentioned in the paper, I cut it to between 0.5 and 10.1 seconds and learned that there is no OOM error.

[Tomiinek]

Oh yes, good point! 👍 Thanks.

[dina-adel]

I encountered the same error while training on Arabic data; however, the length of audios is within the range. Any suggestions?

[Tomiinek]

Hello, what size are your GPUs? Do you also use 2 x 8GB? Have you changed some parameters such as batch size or some audio-related stuff? What about the maximal length of your transcripts?

[divineSix]

I'm trying to train this model on my data and I'm getting a CUDA OOM error during the second epoch. The first epoch is completed successfully, and the second epoch has 20% of the batches processed when this fails. My setup is 4x 12GB 1080s.

My dataset is of 6 languages, and I'm using the default multi-lingual hyperparameters with the exception of batch size, which is 24. My training dataloader has 2375 batches.

I've been trying to solve this for a while now and will summarize my attempts below:

• Tried it with a very small version of my dataset (dataloader with 30 batches), and did not run into any CUDA issues early on. Seems like there are memory artifacts that are not being released every iteration, so with sufficiently long enough steps they're accumulating
• **This leads me to believe that there are memory artifacts that are accumulating every iteration/step. ** i.e.