import deepwave error about deepwave

[ar4]

Hi Naveen, Thank you for writing. The only errors that I see in that output seem to be ones like "Could not open output file '/localscratch/1139367/tmpxft_00001ff3_0000000a'". The compiler appears to want to store temporary files in the /localscratch directory and my guess is that it is unable to. Is it possible that it does not have permission to write to that location, or might /localscratch be full?

[ar4]

Hi Naveen, You might like to try setting the TMPDIR environment variable to a location that nvcc will definitely have write access to (such as a directory you create in your home directory), as the NVCC compiler will apparently then use that location for its temporary files. So you could try "mkdir /home/naveengupta/nvcctmp" and then "export TMPDIR=/home/naveengupta/nvcctmp; python -c 'import deepwave'" (assuming you are using bash). Does that work?

[naveenguptaiitr]

Hi Alan,

I tried this command and still getting the same error. Could you please suggest if you have any insights?

[XuVV]

Hi, Alan.
Thanks for the deepwave.
I also have some problem installing the newest version of deepwave. I used the command " conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=10.2 -c pytorch" to install pytorch and then used "pip install deepwave" . But when I tried to "import deep wave", I have the error below. So could you please tell me which version of pytorch you used?:

Traceback (most recent call last):
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1808, in _run_ninja_build
subprocess.run(
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/nas/home/wxu/lsrtmdip/lsrtm_test_Marmousi.py", line 4, in
import deepwave
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/deepwave/init.py", line 26, in
torch.utils.cpp_extension.load(
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1202, in load
return _jit_compile(
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1425, in _jit_compile
write_ninja_file_and_build_library(
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1537, in write_ninja_file_and_build_library
run_ninja_build(
File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1824, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'deepwave': [1/5] c++ -MMD -MF scalar.o.d -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -march=native -Ofast -fopenmp -c /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/deepwave/scalar.cpp -o scalar.o
[2/5] c++ -MMD -MF scalar_born.o.d -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -march=native -Ofast -fopenmp -c /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/deepwave/scalar_born.cpp -o scalar_born.o
[3/5] /nas/home/wxu/miniconda3/envs/lsrtmdip/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/deepwave/scalar.cu -o scalar.cuda.o
[4/5] /nas/home/wxu/miniconda3/envs/lsrtmdip/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/TH -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/include/THC -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include -isystem /nas/home/wxu/miniconda3/envs/lsrtmdip/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/deepwave/scalar_born.cu -o scalar_born.cuda.o
[5/5] c++ scalar.o scalar_born.o scalar.cuda.o scalar_born.cuda.o -shared -march=native -Ofast -fopenmp -L/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/nas/home/wxu/miniconda3/envs/lsrtmdip/lib64 -lcudart -o deepwave.so
FAILED: deepwave.so
c++ scalar.o scalar_born.o scalar.cuda.o scalar_born.cuda.o -shared -march=native -Ofast -fopenmp -L/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/nas/home/wxu/miniconda3/envs/lsrtmdip/lib64 -lcudart -o deepwave.so
/usr/bin/ld: cannot find -lcudart: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Process finished with exit code 1

[ar4]

@naveenguptaiitr That is a pity. Can you post the new error message? If it is indeed exactly the same - still trying to write to /localscratch - then it seems that NVCC is not using (or receiving) the updated TMPDIR environment variable. If it is using the new TMPDIR and is still unable to write then there seems to be a different problem. Can you also report what the output is when you run the command "touch $TMPDIR/naveentest" on the system that you are trying to use Deepwave on?

[naveenguptaiitr]

I am able to fix the issue with the import using the commands you shared first. The problem was with the updating the TMPDIR variable. Thank you very much for your suggestion and the import issue is fixed.

However, when I am running the FWI inversion on marmousi data, I am getting Location not within the model error. Could you share your insights on this? Thank you again for quickly helping out with the issue.

[ar4]

I am glad to hear that you are now able to use Deepwave. Are you able to show the code you are using that results in an error? Location not within the model means that one or more of the source or receiver locations are outside the velocity model.

[naveenguptaiitr]

Hi Alan,
I was trying to figure out the issue myself and try to understand it, however, I am unable to solve it. I am the using the following code for predicting amplitude for a given velocity model. Please share how to fix the problem. Thank you very much for helping out.!

####################################################
from pysit import *
from pysit.gallery import marmousi2

import torch
import torch.nn as nn
import deepwave
from deepwave import scalar

import numpy as np
import matplotlib.pyplot as plt

device = torch.device('cuda')

Marmousi velocity model

The velocity model size is (96, 192) -> (nz, ny)

vp_true, vp0, m, d = marmousi2(origin=(7500, 355),
size=(1910, 950),
pixel_scale="small",
initial_model_style="gradient",
initial_config={'min':1500.0,'max':3000.0})

vp_true = torch.tensor(vp_true)

acquisition setup

dx = m.deltas[0]
nz, ny = vp_true.shape
num_dims = 2

num_shots = 20

num_sources_per_shot = 1
source_spacing = ny*dx / num_shots #96.0

num_receivers_per_shot = 96
receiver_spacing = ny*dx / num_receivers_per_shot #20.0

peak_freq = 15
dt = 0.004 # 4ms

n_seconds = 4
nt = int(n_seconds / dt)

source_locations

x_s = torch.zeros(num_shots, num_sources_per_shot, num_dims)
x_s[:, 0, 1] = torch.arange(num_shots).float() * source_spacing

receivers_locations

x_r = torch.zeros(num_shots, num_receivers_per_shot, num_dims)
x_r[0, :, 1] = torch.arange(num_receivers_per_shot).float() * receiver_spacing
x_r[:, :, 1] = x_r[0, :, 1].repeat(num_shots, 1)

#source amplitude tensor
source_amplitudes = (deepwave.wavelets.ricker(peak_freq, nt, dt, 1/peak_freq).repeat(num_shots, num_sources_per_shot,1))

print(source_amplitudes.shape)

predict the amplitude on the marmousi velocity model

receiver_amplitudes_true = scalar(vp_true, dx, dt, source_amplitudes=source_amplitudes,
source_locations=x_s, receiver_locations=x_r,
accuracy=8,pml_freq=peak_freq)

#################################################################

Error -

---

RuntimeError Traceback (most recent call last)
Cell In[1], line 60
58 print(source_amplitudes.shape)
59 # predict the amplitude on the marmousi velocity model
---> 60 receiver_amplitudes_true = scalar(vp_true, dx, dt, source_amplitudes=source_amplitudes,
61 source_locations=x_s, receiver_locations=x_r,
62 accuracy=8,pml_freq=peak_freq)

File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.py:360, in scalar(v, grid_spacing, dt, source_amplitudes, source_locations, receiver_locations, accuracy, pml_width, pml_freq, max_vel, survey_pad, wavefield_0, wavefield_m1, psiy_m1, psix_m1, zetay_m1, zetax_m1, origin, nt, model_gradient_sampling_interval, freq_taper_frac, time_pad_frac)
125 def scalar(v: Tensor,
126 grid_spacing: Union[int, float, List[float],
127 Tensor],
(...)
147 Tensor, Tensor,
148 Tensor]:
149 """Scalar wave propagation (functional interface).
150
151 This function performs forward modelling with the scalar wave equation.
(...)
353
354 """
355 (models, source_amplitudes_l, wavefields,
356 pml_profiles, sources_i_l, receivers_i_l,
357 dy, dx, dt, nt, n_batch,
358 step_ratio, model_gradient_sampling_interval,
359 accuracy, pml_width_list) =
--> 360 setup_propagator([v], 'scalar', grid_spacing, dt,
361 [wavefield_0, wavefield_m1, psiy_m1, psix_m1,
362 zetay_m1, zetax_m1],
363 [source_amplitudes],
364 [source_locations], [receiver_locations],
365 accuracy, pml_width, pml_freq, max_vel,
366 survey_pad,
367 origin, nt, model_gradient_sampling_interval,
368 freq_taper_frac, time_pad_frac)
369 v = models[0]
370 wfc, wfp, psiy, psix, zetay, zetax = wavefields

File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:72, in setup_propagator(models, prop_type, grid_spacing, dt, wavefields, source_amplitudes, source_locations, receiver_locations, accuracy, pml_width, pml_freq, max_vel, survey_pad, origin, nt, model_gradient_sampling_interval, freq_taper_frac, time_pad_frac)
70 dy, dx = set_dx(grid_spacing)
71 pad = [fd_pad + width for width in pml_width]
---> 72 models, locations = extract_survey(
73 models,
74 source_locations + receiver_locations,
75 survey_pad, wavefields, origin, pml_width
76 )
77 source_locations = locations[:len(source_locations)]
78 receiver_locations = locations[len(source_locations):]

File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:584, in extract_survey(models, locations, survey_pad, wavefields, origin, pad)
576 def extract_survey(models: List[Tensor],
577 locations: List[Optional[Tensor]],
578 survey_pad: Optional[Union[int,
(...)
582 pad: List[int]) -> Tuple[List[Tensor],
583 List[Optional[Tensor]]]:
--> 584 check_locations_are_within_model(models[0].shape, locations)
585 if any([wavefield is not None for wavefield in wavefields]):
586 survey_extents = get_survey_extents_from_wavefields(
587 wavefields, origin, pad
588 )

File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:610, in check_locations_are_within_model(model_shape, locations)
608 for dim, model_dim_shape in enumerate(model_shape):
609 if location[..., dim].max() >= model_dim_shape:
--> 610 raise RuntimeError("Locations must be within model.")

RuntimeError: Locations must be within model.

[ar4]

Hi Naveen,

The problem is that your velocity model seems to be shaped incorrectly. When I run your code, the output of print(vp_true.shape, nz, ny) is torch.Size([18432, 1]) 18432 1. Your model is thus a 1D vector rather than a 2D model. If I add the line

vp_true = vp_true.reshape(96, 192)

then it will have the correct shape.

Furthermore, Deepwave uses cell indexes as source/receiver locations, so you also need to remove the multiplication by dx when setting your locations, so spacings become

source_spacing = ny / num_shots
receiver_spacing = ny / num_receivers_per_shot

If you need to have locations that are not centered on grid cells then you may wish to look at Deepwave's location interpolation example.

After making these changes I was able to run your code, and produced an image of first shot's data using

vmin, vmax = np.percentile(receiver_amplitudes_true[-1][0], [5,95])
plt.imshow(receiver_amplitudes_true[-1][0].T, aspect='auto', vmin=vmin, vmax=vmax)

[naveenguptaiitr]

Hi Alan,

Thank you very much for your help.! This is working and am delighted to get this working. Appreciate your help. I had few more questions to ask. Could you please address following questions?

1. What is store in the receiver_amplitudes_true tuple from 0 to 5 index?
2. Can we not run the forward model for multiple velocity models at once?

Thank you again for helping out with the issue. :)

[ar4]

Hi Naveen,

That is good news.

The Usage page of the Deepwave documentation describes all of the inputs and outputs. The other outputs relate the the wavefield at the final timestep. One of the reasons that these are provided is so that calls to Deepwave can be chained together, allowing you to continue propagating to later timesteps (as in the checkpointing example).

On its own, Deepwave does not provide a way to propagate with different velocity models simultaneously, but you might be able to get that to work by combining it with something else, such as by using Python's or PyTorch's multiprocessing capabilities to run Deepwave in multiple processes.

[naveenguptaiitr]

Hi Alan,

Thank you very much for sharing your insights. I will check out the documentation to learn more about the deepwave. It is a great library and indeed super helpful for seismic inversion problems. Thank you for helping out.