CPU Bottleneck? about multilingual_text_to_speech

[Tomiinek]

There are no extra CPU-expensive operations during training. It has to load a few spectrograms from disk to construct a batch and move it to GPU memory (note that the train.py script has the loader_workers parameter).

When training on two 8GB GTX1080s with batch size 64 (in parallel), I get the GPU utilization fluctuating between 50% and 94% Smaller batches result in more memory transfers and your GPU probably waits more often.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leijue222]

(note that the train.py script has the loader_workers parameter).

When I trained for 9 hours, the epoch was 22, and the step was 8k, the process was interrupted by the error of：RuntimeError: DataLoader worker (pid 16904) is killed by signal: Aborted.

Therefore, I restore training from a checkpoint and set loader_workers=0.
I hope the training will not be interrupted anymore.