tjyuyao / ice-learn Goto Github PK

Pickling in multiprocess may trigger the following error:

INTERNAL ASSERT FAILED at "../aten/src/ATen/MapAllocator.cpp":263, please report a bug to PyTorch. 
unable to open shared memory object </torch_10722_1002> in read-write mode

According to this issue, I use ulimit -n 4096 and the problem fixed.

Commit 7f6741e implemented a dist_backend argument for ElasticLauncher, that supports ["nccl", "gloo", "mpi", "auto"]. If given "auto", will use "nccl" for "cuda" device and "gloo" for "cpu" device in general. If cuda devices has duplicates, e.g. "cuda:0,0", then "gloo" will be used as "nccl" does not support that and trigger issue similar to Lightning-AI/pytorch-lightning#4420 (comment) .

ice will ensure only use broadcast and all_reduce function, which is available for both gloo and nccl backend on cuda devices, according to https://pytorch.org/docs/stable/distributed.html#backends .

This could happen on some machines when the environmental variables are not properly configured.

export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export C_INCLUDE_PATH=${CUDA_HOME}/include:${C_INCLUDE_PATH}
export LIBRARY_PATH=${CUDA_HOME}/lib64:$LIBRARY_PATH
pip install pycuda