DataLoader worker is killed in Docker about torchmetrics

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Hi @karwojan, thanks for reporting this issue.
I cannot really comprehend what is going on here. My gut feeling is that this has nothing to do with torchmetrics and is more of a torch issue, but who knows. I tried it on my own machine and I could not really reproduce the issue.
Could you maybe report what happens if you run the code with the CUDA_LAUNCH_BLOCKING env set as suggested in the error message?

[karwojan]

Hi @SkafteNicki, thanks for your answer.
Have you tried to run this script in docker using the following command?

docker run --rm -it --gpus=all -v `pwd`:/error pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime /bin/bash -c "pip install torchmetrics==1.4.0 && python /error/error.py"

Without Docker, I am also not able to reproduce the issue.
When I run this script with the CUDA_LAUNCH_BLOCKING env, result is exactly the same.

[SkafteNicki]

Hi @karwojan,
I tried some more and was finally able to reproduce the issue.
I then ran git bisect to narrow down when bug was introduced, with v1.3.2 set as a good commit and v1.4.0 set as a bad commit. In the end the bisecting resulted in this commit cd7ccfc from merging this PR #2468 being marked as the starting point of the bug.
I am still not sure what change in the PR actually is causing this, but I try to narrow it down.