cannot find -lcudart about deepwave

[XuVV]

Doctor, I am the one who should say "Thank you"!
Thank you very for such great contributions. And also thanks for your kindness.

[XuVV]

Hello, thanks, Alan. This is the output :
ls: cannot access '/home/wxu/miniconda3/envs/lsrtmdip/lib64/libcudart*': No such file or directory
ls: cannot access '/home/wxu/miniconda3/envs/lsrtmdip/lib/libcudart*': No such file or directory

[XuVV]

Hello, Alan.
When I used this version of pytorch "conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch" and this version of deepwave "pip install deepwave==0.0.8", it worked.
So could you please tell me which version of pytorch do you use when you use the newest version of deepwave?

[ar4]

Hmm. The CUDA library doesn't seem to be where it expects it to be. Can you try running "echo $CUDA_HOME" when you are in the Conda environment you are trying to install Deepwave in, to see what that environment variable is set to?

Is there a reason why you are trying to use an old version of CUDA? If you just use the current PyTorch install command ("conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia") in a new Conda environment, and then try to install Deepwave, does it work?

I am currently using PyTorch 1.12.1.

[XuVV]

Hello, Alan.
When I tried to run "echo $CUDA_HOME", it outputs nothing. I don't know why.
There is no special reason I used an old version of CUDA, just followed someone else.
When I try to use the current PyTorch install command ("conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia") in a new Conda environment, and then try to install Deepwave, it doesn't work still, which has the error:
Traceback (most recent call last):
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
subprocess.run(
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/nas/home/wxu/lsrtmdip/lsrtm_test_Marmousi.py", line 4, in
import deepwave
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/init.py", line 31, in
torch.utils.cpp_extension.load(
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return jit_compile(
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1508, in jit_compile
write_ninja_file_and_build_library(
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1623, in write_ninja_file_and_build_library
run_ninja_build(
File "/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1916, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'deepwave': [1/7] /nas/home/wxu/miniconda3/envs/lsrtmdipnew/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/scalar.cu -o scalar.cuda.o
[2/7] /nas/home/wxu/miniconda3/envs/lsrtmdipnew/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/elastic.cu -o elastic.cuda.o
/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/elastic.cu(1613): warning #177-D: function "::zero_edge_bottom" was declared but never referenced

/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/elastic.cu(1618): warning #177-D: function "::zero_edge_left" was declared but never referenced

[3/7] /nas/home/wxu/miniconda3/envs/lsrtmdipnew/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/scalar_born.cu -o scalar_born.cuda.o
[4/7] c++ -MMD -MF scalar.o.d -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -march=native -Ofast -fopenmp -c /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/scalar.cpp -o scalar.o
[5/7] c++ -MMD -MF elastic.o.d -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -march=native -Ofast -fopenmp -c /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/elastic.cpp -o elastic.o
[6/7] c++ -MMD -MF scalar_born.o.d -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdipnew/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -march=native -Ofast -fopenmp -c /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/deepwave/scalar_born.cpp -o scalar_born.o
[7/7] c++ scalar.o scalar_born.o elastic.o scalar.cuda.o scalar_born.cuda.o elastic.cuda.o -shared -march=native -Ofast -fopenmp -L/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib64 -lcudart -o deepwave.so
FAILED: deepwave.so
c++ scalar.o scalar_born.o elastic.o scalar.cuda.o scalar_born.cuda.o elastic.cuda.o -shared -march=native -Ofast -fopenmp -L/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib64 -lcudart -o deepwave.so
/usr/bin/ld: cannot find -lcudart: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Process finished with exit code 1

[ar4]

Still the same problem of not finding the CUDA libraries. Did you get any warning messages when running the PyTorch install command (which I think should also install the CUDA libraries)?

Since $CUDA_HOME is not set, we can see where PyTorch thinks CUDA is installed. I think this should do it (run it in the new environment you installed PyTorch in):
python -c "from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)"

[XuVV]

yes, when I runed this python -c "from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)"
The output is "/nas/home/wxu/miniconda3/envs/lsrtmdipnew".

[XuVV]

when I did "export CUDA_HOME=/nas/home/wxu/miniconda3/envs/lsrtmdipnew", and then tried to import deepwave, there is still the same problem.

[ar4]

OK. Can you try to find libcudart.so in that directory ("/nas/home/wxu/miniconda3/envs/lsrtmdipnew")? It's usually in the "lib64" subdirectory of it, but compiling Deepwave is failing because apparently it is not there. I have seen instances before where it is instead in the "lib" subdirectory, but when we searched before (in a different environment, but the result will probably be the same), it wasn't there either. Going to that directory and running the command find . -name "libcudart*" might help to locate it.

Also, just to confirm: you have an appropriate GPU, and didn't get any error messages when installing PyTorch?

[XuVV]

Doctor:
In this directory ("/nas/home/wxu/miniconda3/envs/lsrtmdipnew"), there is no "lib64" subdirectory but "lib", so when I runned the command find . -name "libcudart*", in the "lib" directory, it shows that:
./libcudart.so
./libcudart.so.11.0
./libcudart_static.a
./libcudart.so.11.6.55

When I installed PyTorch, it only showed that:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: done

[ar4]

OK, so in this environment there is a libcudart.so in the "lib" directory. That is promising! I think you will thus solve your problem if you create a softlink from "lib64" to "lib". You can do that with this command: ln -s /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib /nas/home/wxu/miniconda3/envs/lsrtmdipnew/lib64" (or, if it is easier, go to /nas/home/wxu/miniconda3/envs/lsrtmdipnew and then run ln -s lib lib64). You can then try importing Deepwave again, and hopefully it will work this time.

[XuVV]

Thank you very much, Doctor. It WORKED!
But I still have a stupied question: do I set the "export CUDA_HOME=/nas/home/wxu/miniconda3/envs/lsrtmdipnew" wrong?
How can I use the cuda in the conda environment?

[ar4]

That is good news! Thank you for your patience. Hopefully you will find Deepwave useful enough to be worth the effort.

I see now that we could have solved this much earlier - I made a mistake in the first command I asked you to run. I wrote ls /home/wxu/miniconda3/envs/lsrtmdip/lib/libcudart*, but I should have written ls /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/libcudart*. I'm sorry!

There is no need for you to set CUDA_HOME. I don't think the problem was caused by you. For some reason the CUDA libraries are sometimes installed in the "lib" directory, but PyTorch is hardcoded to expect them in the "lib64" directory. I created an Issue in the PyTorch repository about it a few months ago as this same problem occurred with someone else, but it seems that it has not been fixed yet. I think your CUDA should work correctly - we just created a link from the "lib64" directory that PyTorch expects to the "lib" one where the libraries are actually installed, so PyTorch is able to find them.