Some issues in training about visorgpt

[Sierkinhane]

Hi, I did not encounter such a problem, but from the error information, it seems that you should put all the tensors on the same device rather that some on cuda and some on cpu.

[VicZlq]

Thank you very much for your reply!
I did the dubug and in trainer.py line 56:
if args.deepspeed:
worker(args.local_rank, None, args, model_for_training, model_for_dataloader)
args.local_rank=None,

This resulted in a subsequent line 559 where the:
# Get logger
args.logger = init_logger(args)
if args.deepspeed:
import deepspeed
deepspeed.init_distributed(dist_backend=args.backend)
rank = dist.get_rank()
gpu_id = proc_id
elif args.dist_train:
rank = gpu_ranks[proc_id]
gpu_id = proc_id

gpu_id = proc_id=None

Here are my startup command parameters
"--deepspeed",
"--deepspeed_config", "/storage/zhaoliuqing/code/VisorGPT/train/models/deepspeed_config.json",
"--dataset_path", "/storage/zhaoliuqing/code/VisorGPT/train/visorgpt_dagger_train_seq.pt",
"--vocab_path", "/storage/zhaoliuqing/code/VisorGPT/train/models/google_uncased_en_coord_vocab.txt",
"--config_path", "/storage/zhaoliuqing/code/VisorGPT/train/models/gpt2/config.json",
"--output_model_path", "/storage/zhaoliuqing/code/VisorGPT/train/models/visorgpt_dagger_train_seq.bin",
"--world_size", "2",
"--gpu_ranks", "0","1",
"--total_steps", "200000",
"--save_checkpoint_steps", "10000",
"--report_steps", "100",
"--learning_rate", "5e-5",
"--batch_size", "16"
I only changed the values for world size and gpu_ranks.