indices should be either on cpu or on the same device as the indexed tensor (cpu) about pathml

[WilliWespe]

Yes, I did something similar to this. In my case I explicitly wanted the computations to be on the gpu. I edited the utils.py file in the pathml package like this:

In line 73 write: true_1_hot = torch.eye(num_classes + 1).to('cuda:0')[true.squeeze(1)]

You also have to add .to('cuda:0') to the tensor in line 84 (the else block).

Until now the program runs. I couldn't test a full training so far, because it needs to long. (17 hours for a full training. Is this normal?)

But the outer for loop ran for two iterations without error.

[WilliWespe]

I have the exact same problem. I also run it on HPC but with 2 GPUs, each having 15G memory.
I think it has something to do with the newest torch version 1.13.0: pytorch/pytorch#90194

So far I couldn't resolve the problem.
Did you (or anybody else) find a solution?

[jacob-rosenthal]

Is this happening because torch.eye() is created on cpu but the true array has already been sent to the device? If so, maybe it would help to create the torch.eye on the same device, e.g. something like: true_1_hot = torch.eye(num_classes + 1, device = device)[true.squeeze(1)]. Or, you could try sending them all to cpu, something like true_1_hot = torch.eye(num_classes + 1)[true.cpu().squeeze(1)]

[luzy05111036]

@WilliWespe how to edit the utils.py in pathml? I clone the pathml to hpc, then edit the utils.py, and use the second method to install pathml. But, there are several errors.

[WilliWespe]

Well, in my case i first changed my directory to miniconda3/envs/py38/lib/python3.8/site-packages/pathml/ml.
Then I opened utils.py with nano and edited/saved it. After that restart your notebook.

Was this your question?

[luzy05111036]

@WilliWespe thanks. Yes, that's my question and it works. I am starting training now.

[luzy05111036]

problem solved

[FerranC96]

Thank you all for the discussion above!
Can confirm the patch above still works in 2024 with pathml 2.1.1