uoa-cares / essential-gpu-docker Goto Github PK

xception has occurred: RuntimeError
DataLoader worker (pid(s) 4648) exited unexpectedly
RuntimeError: DataLoader worker (pid 4648) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

  File "/workspace/GISLR-torch/run_train_fold0.py", line 289, in run_train
    for t, batch in enumerate(train_loader):
  File "/workspace/GISLR-torch/run_train_fold0.py", line 372, in <module>
    run_train()
RuntimeError: DataLoader worker (pid(s) 4648) exited unexpectedly

RuntimeError: unable to write to file </torch_4655_434291219_38>: No space left on device (28)
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/opt/conda/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 366, in reduce_storage
    fd, size = storage._share_fd_cpu_()
RuntimeError: unable to write to file </torch_4753_3315859216_32>: No space left on device (28)

sudo systemctl restart NetworkManager.service.