IndexError: list index out of range about pytorch-correlation-extension

[ClementPinard]

Hello, long story short, you don't have access to CUDA when compiling.

See more info about it here : pytorch/extension-cpp#71 (comment)

[berengueradrian]

Thanks @ClementPinard, I will look at that and try to solve it.

[berengueradrian]

Hi @ClementPinard, I changed the daemon.json from /etc/docker/ but now I came back with an error I had before:

 => ERROR [14/21] RUN pip install spatial-correlation-sampler                                                                                                                                                                                                                                                                                                        8.3s 
------                                                                                                                                                                                                                                                                                                                                                                    
 > [14/21] RUN pip install spatial-correlation-sampler:                                                                                                                                                                                                                                                                                                                   
0.905 Collecting spatial-correlation-sampler                                                                                                                                                                                                                                                                                                                              
0.982   Downloading spatial_correlation_sampler-0.4.0.tar.gz (9.3 kB)                                                                                                                                                                                                                                                                                                     
0.998   Preparing metadata (setup.py): started                                                                                                                                                                                                                                                                                                                            
3.318   Preparing metadata (setup.py): finished with status 'done'                                                                                                                                                                                                                                                                                                        
3.328 Requirement already satisfied: torch>=1.1 in /miniconda/lib/python3.11/site-packages (from spatial-correlation-sampler) (2.1.0)
3.338 Requirement already satisfied: numpy in /miniconda/lib/python3.11/site-packages (from spatial-correlation-sampler) (1.26.2)
3.384 Requirement already satisfied: filelock in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (3.13.1)
3.385 Requirement already satisfied: typing-extensions in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (4.9.0)
3.387 Requirement already satisfied: sympy in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (1.12)
3.388 Requirement already satisfied: networkx in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (3.2.1)
3.389 Requirement already satisfied: jinja2 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (3.1.2)
3.391 Requirement already satisfied: fsspec in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (2023.12.2)
3.393 Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (12.1.105)
3.395 Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (12.1.105)
3.398 Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (12.1.105)
3.400 Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (8.9.2.26)
3.402 Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (12.1.3.1)
3.405 Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (11.0.2.54)
3.407 Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (10.3.2.106)
3.410 Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (11.4.5.107)
3.412 Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (12.1.0.106)
3.414 Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (2.18.1)
3.417 Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (12.1.105)
3.419 Requirement already satisfied: triton==2.1.0 in /miniconda/lib/python3.11/site-packages (from torch>=1.1->spatial-correlation-sampler) (2.1.0)
3.434 Requirement already satisfied: nvidia-nvjitlink-cu12 in /miniconda/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.1->spatial-correlation-sampler) (12.3.101)
3.525 Requirement already satisfied: MarkupSafe>=2.0 in /miniconda/lib/python3.11/site-packages (from jinja2->torch>=1.1->spatial-correlation-sampler) (2.1.3)
3.554 Requirement already satisfied: mpmath>=0.19 in /miniconda/lib/python3.11/site-packages (from sympy->torch>=1.1->spatial-correlation-sampler) (1.3.0)
3.573 Building wheels for collected packages: spatial-correlation-sampler
3.574   Building wheel for spatial-correlation-sampler (setup.py): started
5.875   Building wheel for spatial-correlation-sampler (setup.py): finished with status 'error'
5.887   error: subprocess-exited-with-error
5.887   
5.887   × python setup.py bdist_wheel did not run successfully.
5.887   │ exit code: 1
5.887   ╰─> [13 lines of output]
5.887       No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
5.887       running bdist_wheel
5.887       /miniconda/lib/python3.11/site-packages/torch/utils/cpp_extension.py:502: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
5.887         warnings.warn(msg.format('we could not find ninja.'))
5.887       running build
5.887       running build_py
5.887       creating build
5.887       creating build/lib.linux-x86_64-cpython-311
5.887       creating build/lib.linux-x86_64-cpython-311/spatial_correlation_sampler
5.887       copying Correlation_Module/spatial_correlation_sampler/spatial_correlation_sampler.py -> build/lib.linux-x86_64-cpython-311/spatial_correlation_sampler
5.887       copying Correlation_Module/spatial_correlation_sampler/__init__.py -> build/lib.linux-x86_64-cpython-311/spatial_correlation_sampler
5.887       running build_ext
5.887       error: [Errno 2] No such file or directory: '/usr/local/cuda/bin/nvcc'
5.887       [end of output]
5.887   
5.887   note: This error originates from a subprocess, and is likely not a problem with pip.
5.887   ERROR: Failed building wheel for spatial-correlation-sampler
5.887   Running setup.py clean for spatial-correlation-sampler
8.188 Failed to build spatial-correlation-sampler
8.189 ERROR: Could not build wheels for spatial-correlation-sampler, which is required to install pyproject.toml-based projects
------
Dockerfile:71
--------------------
  69 |     
  71 | >>> RUN pip install spatial-correlation-sampler
  72 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install spatial-correlation-sampler" did not complete successfully: exit code: 1

[ClementPinard]

It's weird because it tells it does not have access to the nvss program (the cuda compiler) , but you're using a devel docker image, which should feature nvcc

See more info for your problem : NVIDIA/nvidia-docker#1160

Try to call nvcc from outside the pip command to see if nvcc is indeed reachable.

[berengueradrian]

Hi @ClementPinard, I finally solved it by adding these lines:

ENV PATH=${CUDA_HOME}/bin:${PATH}
ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}

Thanks a lot for the help.