About Running Issues on NVIDIA RTX 4090 Graphics Card about minkowskiengine

[XuyangZhang0223]

I also encounter this error when using my RTX 4060 graphics card, the error is the same as yours (MinkowskiEngine/src/gpu.cu:100) when I set the SparseTensorQuantizationMode to UNWEIGHTED_AVERAGE, but there is no error when it is set to RANDOM_SUBSAMPLE.

Have you find the solution? Thx!

[ZiliangMiao]

I also encounter the same error using RTX 4090
RuntimeError: at /tmp/pip-req-build-up7naarj/src/gpu.cu:100

[ZiliangMiao]

@chrischoy Same problem still unsolved, could you please provide any suggestions?
RTX4090, python=3.8, pytorch=1.10.0-cu111, pytorch-lightning=1.9.0, MinkowskiEngine=0.5.4
If quantization_mode=ME.SparseTensorQuantizationMode.UNWEIGHTED_AVERAGE, it will cause RuntimeError:

  File "/home/user/Projects/MosPretrain/scripts/train.py", line 101, in <module>
    main()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/user/Projects/MosPretrain/scripts/train.py", line 97, in main
    trainer.fit(model, train_dataloader, val_dataloader, ckpt_path=resume)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
    return function(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run_train
    self._run_sanity_check()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1267, in _run_sanity_check
    val_loop.run()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 359, in validation_step
    return self.model(*args, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 110, in forward
    return self._forward_module.validation_step(*inputs, **kwargs)
  File "/home/user/Projects/MosPretrain/src/mos4d/models/nusc_models.py", line 88, in validation_step
    out = self.forward(point_clouds)
  File "/home/user/Projects/MosPretrain/src/mos4d/models/nusc_models.py", line 61, in forward
    out = self.model(past_point_clouds)
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/Projects/MosPretrain/src/mos4d/models/nusc_models.py", line 200, in forward
    sparse_tensor = tensor_field.sparse()
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiTensorField.py", line 354, in sparse
    features = MinkowskiSPMMAverageFunction().apply(
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/sparse_matrix_functions.py", line 183, in forward
    result, COO, vals = spmm_average(
  File "/home/user/anaconda3/envs/4dmos/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/sparse_matrix_functions.py", line 93, in spmm_average
    result, COO, vals = MEB.coo_spmm_average_int32(
RuntimeError: <unknown> at /home/user/Installations/MinkowskiEngine/src/gpu.cu:100