Can you resume training at breakpoints? about transgan

[mapengsen]

Ok, thank you very much for your reply. I'll study it carefully and then try it. Thank you very much.

[asarigun]

Hi @mapengsen,

I didn't use any breakpoints during training; however, you can do the following at train.py from this blog which I directly put here. I hope this will be helpful for you;

First set two parameters:

start_epoch: The number of epochs to start training during renewal
resume: Whether to implement breakpoint resume training

start_epoch = 0
resume = True

After the model and optimizer are set up, add the following code, load the checkpoint and then train,

if opt.resume:
   if os.path.isfile('checkpoint'):
       checkpoint = torch.load('checkpoint')
       start_epoch = checkpoint['epoch'] + 1
       model.load_state_dict(checkpoint['model'])
       optimizer.load_state_dict(checkpoint['optimizer'])
       print("=> loaded checkpoint (epoch {})".format(checkpoint['epoch']))
   else:
       print("=> no checkpoint found")

After each round of training and saving the model, add the following code to save the epoch, model, and optimizer information to the checkpoint file.

checkpoint = {
            'epoch': epoch,
            'model': model.state_dict(),
            'optimizer': optimizer.state_dict(),
        }
        torch.save(checkpoint,'checkpoint')

This method also supports the tensorboardX visualized breakpoint continuation graph, but it will generate an extra log file, which only records the situation after the breakpoint. The data after the breakpoint will continue to be recorded in the original log file.