GPU problem about minkowskiengine

[chrischoy]

Seems related to this #40

[chrischoy]

This should not happen for sure. I'll dig deeper into this after some deadlines.

But is the GPU mapping different from the physical GPU ID on your system?
I found that on some servers the GPU ID and the actual GPU used are mapped differently.

Also, have you tried using export CUDA_VISIBLE_DEVICES=0 or any other GPU ID to mask the visible GPU to the program?

[murrdpirate]

Edit: I did just try setting export CUDA_VISIBLE_DEVICES=1 as you mentioned, and that does make it work. That's probably a good enough solution for me. See below for original post.

I seem to be having this issue as well. Using the default cuda device works without issues.

I'm running:

• MinkowskiEngine 0.4.3
• cuda 10.2
• python 3.7.7
• pytorch 1.5.0

I'm actually running FCGF, but I figured this was the best place to mention the issue.

Here's the output (basically same as @GaoQiyu ):

Traceback (most recent call last):
File "test.py", line 80, in
main(config)
File "test.py", line 55, in main
trainer._valid_epoch()
File "/home/km/workspaces/FCGF/lib/trainer.py", line 338, in _valid_epoch
F0 = self.model(sinput0).F
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/km/workspaces/FCGF/model/resunet.py", line 143, in forward
out_s1 = self.conv1(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/MinkowskiEngine/MinkowskiConvolution.py", line 281, in forward
input.coords_man)
File "/usr/local/lib/python3.7/dist-packages/MinkowskiEngine/MinkowskiConvolution.py", line 92, in forward
ctx.coords_man.CPPCoordsManager)
RuntimeError: an illegal memory access was encountered at src/convolution.cu:285

[chrischoy]

Seems like the default cuda calls for cuda malloc and the pytorch default GPU are using different GPU IDs.