Build error when using pytorch 1.7 about pytorch-correlation-extension

[ClementPinard]

Hi, after some investigation on my system :
Ubuntu 20.04
Pytorch 1.7
RTX 2080 Super
Cuda 11.1

I had the same bug, and using an explicitely named namespace instead of a generic one seems to do the trick. I have genuinely no idea, why it works now though, especially when the official extension-cpp repo does build on my machine and has a generic namespace.

Anyway, you can try the change with the branch fix_1.7 . If the build is successful (along with check.py and grad_check.py) I'll merge the changes

Clément

[ClementPinard]

Hi thanks for your input, I'll try to take a look at it as soon as possible (I am already using my computer for some training, and I'll have to wait before testing a new cuda version.

Have you tried the officiel cuda-extension-cpp example ? Does it compile ? (There's is an on going PR I did to fix deprecated calls that you might want to checkout instead of master)

[danqu130]

Thank you for your reply!

I have tried the official example from https://github.com/pytorch/extension-cpp/tree/master/cuda

python setup.py install

Yes, there is a lot of warning about deprecated calls etc. But it can be compiled.

...
creating build/lib.linux-x86_64-3.8
g++ -pthread -shared -B /home1/user/anaconda3/compiler_compat -L/home1/user/anaconda3/lib -Wl,-rpath=/home1/user/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda.o /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda_kernel.o -L/home1/user/anaconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda-11.1/lib64 -lc10 -ltorch
-ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/lltm_cuda.cpython-38-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.8/lltm_cuda.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for lltm_cuda.cpython-38-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/lltm_cuda.py to lltm_cuda.cpython-38.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.lltm_cuda.cpython-38: module references __file__
creating dist
creating 'dist/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing lltm_cuda-0.0.0-py3.8-linux-x86_64.egg
creating /home1/user/anaconda3/lib/python3.8/site-packages/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg
Extracting lltm_cuda-0.0.0-py3.8-linux-x86_64.egg to /home1/user/anaconda3/lib/python3.8/site-packages
Adding lltm-cuda 0.0.0 to easy-install.pth file

Installed /home1/user/anaconda3/lib/python3.8/site-packages/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg
Processing dependencies for lltm-cuda==0.0.0
Finished processing dependencies for lltm-cuda==0.0.0

And I use your fixed branch from https://github.com/ClementPinard/extension-cpp/tree/fix_deprecated/cuda, those warning are greatly reduced and can also compile successfully.

running install
running bdist_egg
running egg_info
creating lltm_cuda.egg-info
writing lltm_cuda.egg-info/PKG-INFO
writing dependency_links to lltm_cuda.egg-info/dependency_links.txt
writing top-level names to lltm_cuda.egg-info/top_level.txt
writing manifest file 'lltm_cuda.egg-info/SOURCES.txt'
reading manifest file 'lltm_cuda.egg-info/SOURCES.txt'
writing manifest file 'lltm_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'lltm_cuda' extension
creating /home1/user/software/pytorch/extension-cpp/cuda/build
creating /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8
Emitting ninja build file /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda.o.d -pthread -B /home1/user/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home1/user/anaconda3/include/python3.8 -c -c /home1/user/software/pytorch/extension-cpp/cuda/lltm_cuda.cpp -o /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=lltm_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home1/user/anaconda3/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:149:0,
                 from /home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                 from /home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                 from /home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                 from /home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:12,
                 from /home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from /home1/user/software/pytorch/extension-cpp/cuda/lltm_cuda.cpp:1:
/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:84:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
 #pragma omp parallel for if ((end - begin) >= grain_size)

[2/2] /usr/local/cuda-11.1/bin/nvcc -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.1/include -I/home1/user/anaconda3/include/python3.8 -c -c /home1/user/software/pytorch/extension-cpp/cuda/lltm_cuda_kernel.cu -o /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=lltm_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/ATen/core/boxing/impl/boxing.h(100): warning: integer conversion resulted in a change of sign
/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/ATen/core/op_registration/op_whitelist.h(39): warning: integer conversion resulted in a change of sign

/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/ATen/core/boxing/impl/boxing.h(100): warning: integer conversion resulted in a change of sign
/home1/user/anaconda3/lib/python3.8/site-packages/torch/include/ATen/core/op_registration/op_whitelist.h(39): warning: integer conversion resulted in a change of sign

creating build/lib.linux-x86_64-3.8
g++ -pthread -shared -B /home1/user/anaconda3/compiler_compat -L/home1/user/anaconda3/lib -Wl,-rpath=/home1/user/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda.o /home1/user/software/pytorch/extension-cpp/cuda/build/temp.linux-x86_64-3.8/lltm_cuda_kernel.o -L/home1/user/anaconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda-11.1/lib64 -lc10 -ltorch
-ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/lltm_cuda.cpython-38-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.8/lltm_cuda.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for lltm_cuda.cpython-38-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/lltm_cuda.py to lltm_cuda.cpython-38.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cuda.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.lltm_cuda.cpython-38: module references __file__
creating dist
creating 'dist/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing lltm_cuda-0.0.0-py3.8-linux-x86_64.egg
removing '/home1/user/anaconda3/lib/python3.8/site-packages/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg' (and everything under it)
creating /home1/user/anaconda3/lib/python3.8/site-packages/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg
Extracting lltm_cuda-0.0.0-py3.8-linux-x86_64.egg to /home1/user/anaconda3/lib/python3.8/site-packages
lltm-cuda 0.0.0 is already the active version in easy-install.pth

Installed /home1/user/anaconda3/lib/python3.8/site-packages/lltm_cuda-0.0.0-py3.8-linux-x86_64.egg
Processing dependencies for lltm-cuda==0.0.0
Finished processing dependencies for lltm-cuda==0.0.0

As I said earlier, I guess it doesn't seem to be a newer CUDA issue, but rather a problem introduced by upgrading to Python 1.7. On a 2080ti computer with CUDA10.1, I can't compile this package with Pytorch 1.7, and the error reports are same with use 3090 and CUDA11.1. But if I go back to use Pytorch 1.6 on this 2080ti and CUDA10.1 computer, I can compile it successfully.

OS: Ubuntu 18.0.4
PyTorch version: 1.7.0+cu110
How you installed PyTorch: pip
Python version: 3.8.3
CUDA/cuDNN version: 11.1
GPU models and configuration: RTX 3090
GCC version (if compiling from source): 7.5.0

OS: Ubuntu 18.0.4
PyTorch version: 1.6.0 and 1.7.0
How you installed PyTorch: pip
Python version: 3.8.3
CUDA/cuDNN version: 10.1
GPU models and configuration: RTX 2080ti
GCC version (if compiling from source): 7.5.0

Looking forward to your results!

[veizgyauzgyauz]

When I compiled on my system with specs
Ubuntu 18.04,
gcc 7.5.0,
GPU RTX 3090,
CUDA 11.1,
Python 3.8,
PyTorch 1.7,
I also got an error about nvcc:
nvcc fatal : Unsupported gpu architecture 'compute_86'

The whole log is

running install
running bdist_egg
running egg_info
writing Correlation_Module/spatial_correlation_sampler.egg-info/PKG-INFO
writing dependency_links to Correlation_Module/spatial_correlation_sampler.egg-info/dependency_links.txt
writing requirements to Correlation_Module/spatial_correlation_sampler.egg-info/requires.txt
writing top-level names to Correlation_Module/spatial_correlation_sampler.egg-info/top_level.txt
reading manifest file 'Correlation_Module/spatial_correlation_sampler.egg-info/SOURCES.txt'
writing manifest file 'Correlation_Module/spatial_correlation_sampler.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/spatial_correlation_sampler
copying Correlation_Module/spatial_correlation_sampler/spatial_correlation_sampler.py -> build/lib.linux-x86_64-3.8/spatial_correlation_sampler
copying Correlation_Module/spatial_correlation_sampler/__init__.py -> build/lib.linux-x86_64-3.8/spatial_correlation_sampler
running build_ext
building 'spatial_correlation_sampler_backend' extension
creating /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8
creating /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module
Emitting ninja build file /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda-11.0/bin/nvcc -DUSE_CUDA -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/TH -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/include/python3.8 -c -c /data/Projects/Pytorch-Correlation-extension/Correlation_Module/correlation_cuda_kernel.cu -o /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=spatial_correlation_sampler_backend -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
FAILED: /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation_cuda_kernel.o 
/usr/local/cuda-11.0/bin/nvcc -DUSE_CUDA -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/TH -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/include/python3.8 -c -c /data/Projects/Pytorch-Correlation-extension/Correlation_Module/correlation_cuda_kernel.cu -o /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=spatial_correlation_sampler_backend -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=sm_86 -std=c++14
nvcc fatal   : Unsupported gpu architecture 'compute_86'
[2/3] c++ -MMD -MF /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation.o.d -pthread -B /data/Dependencies/anaconda3/envs/pytorch1.7/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE_CUDA -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/TH -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/include/python3.8 -c -c /data/Projects/Pytorch-Correlation-extension/Correlation_Module/correlation.cpp -o /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation.o -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=spatial_correlation_sampler_backend -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
[3/3] c++ -MMD -MF /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation_sampler.o.d -pthread -B /data/Dependencies/anaconda3/envs/pytorch1.7/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE_CUDA -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/TH -I/data/Dependencies/anaconda3/envs/pytorch1.7/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/data/Dependencies/anaconda3/envs/pytorch1.7/include/python3.8 -c -c /data/Projects/Pytorch-Correlation-extension/Correlation_Module/correlation_sampler.cpp -o /data/Projects/Pytorch-Correlation-extension/build/temp.linux-x86_64-3.8/Correlation_Module/correlation_sampler.o -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=spatial_correlation_sampler_backend -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
ninja: build stopped: subcommand failed.

[shreyas23]

It builds and the checks (check.py forward/backward and grad_check.py cuda) pass with the following specs:

PyTorch 1.8.0 nightly build
gcc 9.3.0
CUDA 11.1
Python 3.7
RTX 3090, driver: 455.32.00
ubuntu 20.04

[danqu130]

Yes, I've also successfully compiled fix_1.7 branch and passed the checks (check.py forward/backward and grad_check.py cuda).

Thank you for your update!

[PinxueGuo]

Yes, I've also successfully compiled fix_1.7 branch and passed the checks (check.py forward/backward and grad_check.py cuda).

Thank you for your update!

Hello, my friend, I have the same problem, but I can't compile the files under build.Could you please provide these files? I want to install it manually. Thank you!

朋友你好，我也遇到的同样的问题，但我无法编译出build下的文件，请问或许可以提供一下这些文件？我想手动安装一下试试，f非常谢谢！微信：xue937370152

[JackieZhai]

Hi, after some investigation on my system :
Ubuntu 20.04
Pytorch 1.7
RTX 2080 Super
Cuda 11.1

I had the same bug, and using an explicitely named namespace instead of a generic one seems to do the trick. I have genuinely no idea, why it works now though, especially when the official extension-cpp repo does build on my machine and has a generic namespace.

Anyway, you can try the change with the branch fix_1.7 . If the build is successful (along with check.py and grad_check.py) I'll merge the changes

Clément

It helps on RTX 3090 either, thanks!