Expecting cpu when using continue_train function about danspeech_training

[Rasmusafj]

Sorry for late response but was busy because of vacation. :)

The error seems to occur because the model is cast to device (in your case GPU) after the optimizer has been initialized based on the stored state in the model. Hence, the optimizer expects CPU instead of GPU because the model has not been cast to GPU device yet.

I implemented a "hacky" fix that makes sure the model is cast to device after being loaded. I'll make PR in a sec.

Let me know if it works.

Note: I think that the DanSpeechPrimary model from the danspeech library is packaged without "optim_dict", which means that you can't directly continue training (not sure though). I have the model packaged with the optim_dict. Let me know if you need it.