Single-node Multi-GPU training throws CUDA failure: an illegal memory access was encountered. about graphlearn-for-pytorch

[Elnifio]

Additionally, we also tried to use with_gpu=False for glt_dataset.init_node_features, and it throws an IndexError instead of the above CUDA failure when trying to load the first batch of the train dataloader. To reproduce this issue, we follow the same bash command above (using 2 GPUs by default), and update line 250-258 of the above min rep code to:

    glt_dataset.init_node_features(
        node_feature_data=igbh_dataset.feat_dict,
        with_gpu=False,
        # split_ratio=0.15 * min(num_gpus, 4),
        # device_group_list=[
        #     glt.data.DeviceGroup(idx, group) 
        #     for idx, group in enumerate(gpu_groups)
        # ]
    )

And the following error is thrown:

Traceback (most recent call last):
  File "min_rep.py", line 158, in <module>
    torch.multiprocessing.spawn(run,
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/workspace/repository/min_rep.py", line 75, in run
    for batch in train_loader:
  File "/usr/local/lib/python3.8/dist-packages/graphlearn_torch/loader/neighbor_loader.py", line 104, in __next__
    result = self._collate_fn(out)
  File "/usr/local/lib/python3.8/dist-packages/graphlearn_torch/loader/node_loader.py", line 99, in _collate_fn
    x_dict = {ntype : self.data.get_node_feature(ntype)[ids] for ntype, ids in sampler_out.node.items()}
  File "/usr/local/lib/python3.8/dist-packages/graphlearn_torch/loader/node_loader.py", line 99, in <dictcomp>
    x_dict = {ntype : self.data.get_node_feature(ntype)[ids] for ntype, ids in sampler_out.node.items()}
  File "/usr/local/lib/python3.8/dist-packages/graphlearn_torch/data/feature.py", line 145, in __getitem__
    return self.cpu_get(ids)
  File "/usr/local/lib/python3.8/dist-packages/graphlearn_torch/data/feature.py", line 163, in cpu_get
    return self.feature_tensor[ids]
IndexError: index 1004440 is out of bounds for dimension 0 with size 1000000

We have checked the train indices and they are all within 1000000 (min 0, max 599999), so we're not sure where is 1004440 index being yielded.

[husimplicity]

I think this has been solved by #62 . Would you try it again?

[LiSu]

We have fixed the bug in #62 , and added a single-node multi-gpu example.