sunset1995 / directvoxgo Goto Github PK

in lib/cuda/total_variation_kernel.cu

    grad_to_add += (k==0      ? 0 : wz * clamp(param[index]-param[index-1], -1.f, 1.f));
    grad_to_add += (k==sz_k-1 ? 0 : wz * clamp(param[index]-param[index+1], -1.f, 1.f));
    grad_to_add += (j==0      ? 0 : wy * clamp(param[index]-param[index-sz_k], -1.f, 1.f));
    grad_to_add += (j==sz_j-1 ? 0 : wy * clamp(param[index]-param[index+sz_k], -1.f, 1.f));
    grad_to_add += (i==0      ? 0 : wz * clamp(param[index]-param[index-sz_k*sz_j], -1.f, 1.f));
    grad_to_add += (i==sz_i-1 ? 0 : wz * clamp(param[index]-param[index+sz_k*sz_j], -1.f, 1.f));
    grad[index] += grad_to_add;

It seems wx is not used there, maybe it's typos?

I am trying to use DVGO to reconstruct synthetic 3D scenes from the Replica dataset. I gathered an inward facing trajectory of images and poses (generated by pose_spherical with 20 theta angles * 3 phi angles * 3 heights = 180 poses). Since I generate poses first and then use habitat-sim to get the camera view at the poses, I don't have to run colmap. The following is the visualization of camera poses using tools/vis_train.py.

I use the config default_ubn_inward_facing.py with nerfpp dataset. After training, if I use the same training trajectory for testing, DVGO would perfectly render the views. This validates that my image-to-pose mapping is correct. However, if I use an adjested, still inward-facing trajectory for testing, there are distortions as shown bellow (first is rendered by DVGO, second is the ground truth from habitat-sim, they don't have the exact same pose, but show roughly the same view).

My questions:

1. How to improve DVGO's quality and reduce distortions in my case?

• How should I capture a trajectory on which DVGO would better optimize? Is the current trajectory too "regular" (oval-shaped) and that I should add some variations to the poses? Would it help if I simply have a longer trajectory (denser theta angles and heights, wider phi angle range) in the current oval shape?
• What configuration should I tune to reduce the distortions?

2. Also, both the camera bounding box and the white background on the right of the DVGO rendered image seem that DVGO computes a bounding box that doesn't cover the whole room, and that out-of-bounding-box places are not learned and left white. I actually have the exact bounding box computed for the 3D scene. Is there a way to manually set the bounding box for DVGO?

When calculating nearclip_loss in scene_rep_reconstruction, it always seems to 0.

if cfg_train.weight_nearclip > 0:
    near_thres = data_dict['near_clip'] / model.scene_radius[0].item()
    near_mask = (render_result['t'] < near_thres)
    density = render_result['raw_density'][near_mask]
    if len(density):
        nearclip_loss = (density - density.detach()).sum()
        loss += cfg_train.weight_nearclip * nearclip_loss

What was the original intent of that code?

Many thanks to the author for his contribution to this work,
But I'm having some difficulty implementing, which is following:

Using C:\Users\shower\AppData\Local\torch_extensions\torch_extensions\Cache\py39_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\shower\AppData\Local\torch_extensions\torch_extensions\Cache\py39_cu116\adam_upd_cuda\build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Traceback (most recent call last):

File ~\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:1808 in _run_ninja_build
subprocess.run(

File ~\anaconda3\lib\subprocess.py:528 in run
raise CalledProcessError(retcode, process.args,

CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File D:\DirectVoxGO-main\run.py:13 in
from lib import utils, dvgo, dcvgo, dmpigo

File D:\DirectVoxGO-main\lib\utils.py:11 in
from .masked_adam import MaskedAdam

File D:\DirectVoxGO-main\lib\masked_adam.py:8 in
adam_upd_cuda = load(

File ~\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:1202 in load
return _jit_compile(

File ~\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:1425 in _jit_compile
_write_ninja_file_and_build_library(

File ~\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:1537 in _write_ninja_file_and_build_library
_run_ninja_build(

File ~\anaconda3\lib\site-packages\torch\utils\cpp_extension.py:1824 in _run_ninja_build
raise RuntimeError(message) from e

RuntimeError: Error building extension 'adam_upd_cuda'

I have tried many methods including looking for the source of extension_cpp.py or updating and modifying the version of vs and so on, but I can't solve it, I hope to get your help to achieve this work, and I have no experience in compiling so I can‘t sure where the problem is.

Additionally, my environment is Windows11, torch 1.12.0, cuda 11.6 , vs2017 and python 3.9.
Hope this extra configuration information will help you find the problem in my work better, thanks again.

Hi,

I'm trying to run DirectVoxGO using a conda environment with python3.9 and cuda11.3 and with the dependencies listed in the README.md. However, I get the following error:

[2/3] /usr/local/cuda-11/bin/nvcc  -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/include -isystem /users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/include/TH -isystem /users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda-11/include -isystem /users/visics/gkouros/miniconda3/envs/dvgo/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /users/visics/gkouros/projects/nerf-repos/DirectVoxGO/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o 
[3/3] c++ adam_upd.o adam_upd_kernel.cuda.o -shared -L/users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/usr/local/cuda-11/lib64 -lcudart -o adam_upd_cuda.so
Loading extension module adam_upd_cuda...
Traceback (most recent call last):
  File "/users/visics/gkouros/projects/nerf-repos/DirectVoxGO/run.py", line 23, in <module>
    from lib import utils, dvgo, dmpigo, dvgo_video
  File "/users/visics/gkouros/projects/nerf-repos/DirectVoxGO/lib/utils.py", line 11, in <module>
    from .masked_adam import MaskedAdam
  File "/users/visics/gkouros/projects/nerf-repos/DirectVoxGO/lib/masked_adam.py", line 7, in <module>
    adam_upd_cuda = load(
  File "/users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1144, in load
    return _jit_compile(
  File "/users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1382, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "/users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1775, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
ImportError: /users/visics/gkouros/miniconda3/envs/dvgo/lib/python3.9/site-packages/torch/lib/../../../../libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /users/visics/gkouros/.cache/torch_extensions/py39_cu113/adam_upd_cuda/adam_upd_cuda.so)

Any ideas how to solve it?

This work looks promising!
I am thinking if it's feasible to port the CUDA code into JAX and achieve similar speed.
Do you think it's feasible to achieve this?

Hello,

I run the training in a google colab instance.

My code is roughly the following (in different cells):

from google.colab import drive
drive.mount('/content/drive')

%cd "drive/My Drive/DirectVoxGO"

pip install -r requirements.txt

import torch
assert(torch.__version__ == '1.10.0+cu111')

pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu111.html

!python run.py --config configs/blendedmvs/Jade.py --render_test  # train

First error coming out (in the last line of the above), I need ninja:

Using /root/.cache/torch_extensions/py37_cu111 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py37_cu111/adam_upd_cuda...
Traceback (most recent call last):
  File "run.py", line 13, in <module>
    from lib import utils, dvgo
  File "/content/drive/MyDrive/MAREVA_project/DirectVoxGO/lib/utils.py", line 11, in <module>
    from .masked_adam import MaskedAdam
  File "/content/drive/MyDrive/MAREVA_project/DirectVoxGO/lib/masked_adam.py", line 10, in <module>
    verbose=True)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/cpp_extension.py", line 1136, in load
    keep_intermediates=keep_intermediates)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/cpp_extension.py", line 1347, in _jit_compile
    is_standalone=is_standalone)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/cpp_extension.py", line 1418, in _write_ninja_file_and_build_library
    verify_ninja_availability()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/cpp_extension.py", line 1474, in verify_ninja_availability
    raise RuntimeError("Ninja is required to load C++ extensions")
RuntimeError: Ninja is required to load C++ extensions

Then, I do

pip install ninja

Rerunning the line !python run.py --config configs/blendedmvs/Jade.py --render_test # train, I now get a very ugly output, that I put on pastebin because it is too long: https://pastebin.com/Um0cMJfg . Notice that despite the warnings, the training starts at the end.

Is this output expected? If these warnings/errors appear due to PyTorch 1.10.0 and are critical, I would suggest to disable those custom CUDA extensions by default so that training can run on several PyTorch versions, not only 1.8.1.

Thanks a lot!

First of all, thank you for this great work!

I am trying to complement the framework with background computation (similar to plenoxel background approach).
However, I have a hard time understanding the following line of code:

interval = render_kwargs['stepsize'] * self.voxel_size_ratio

In my understanding, the value passed to activate_density() is the distance to the next point which in this case should be closer to something like (interpx[..., 1:] - interpx[..., :-1]) * rays_d.norm(dim=-1, keepdim=True)
But when I passed this set of values, the results are much worse.

Could you explain the rational of the computation of interval?
Thank you in advance,

Best regards,

Pierre

Hi, thanks for your great work. May I know how to understand the below code? thanks a lot.

        rgb_marched = segment_coo(
                src=(weights.unsqueeze(-1) * rgb),
                index=ray_id,
                out=torch.zeros([N, 3]),
                reduce='sum')

For example, I know the size of “src” is [0,3], and the size of "index" is [0], but how to get the result of rgb_marched with the size of "[8192,3]"? thanks.

I see the coarse_train config setting is N_iters = 0 in nerf_unbounded_default.py.
https://github.com/sunset1995/DirectVoxGO/blob/main/configs/nerf_unbounded/nerf_unbounded_default.py#L16
What's the purpose of this setting? It seems we skip the coarse stage.

Another question, how to inspect the fine model training result similar like --export_coarse_only flag?
It seems if we skip the coarse stage, we don't have model to inspect before the fine stage.
Thanks.

@sunset1995 I train nsvf data like bike, palace, the render result for test img is correct

but the render pose(--render_video) is all white img

Hi, thanks a lot for the excellent project!

When I use your nice customized CUDA TV loss to the voxel grid， any advice for the weight of the loss? It seems that the default setting did not use the TV loss, right?

Thanks again!

Thanks for your great work!!! @sunset1995
If the camera pose is to shoot a circle around the object and rotate in a plane, will the bbox be very flat?

Experiments on my own dataset found that the reconstructed objects are also flat.

Thanks for your excellent work. I have trained the models with the given datasets and they are running well. Can you tell me how to run with my own dataset? I processed the multi-view image with colmap

I want to reproduce this work on my platform. It seems the environment is already set up.

(the torch version is 1.12.1+cu113, the gcc version is 7.5.0, and the nvcc version is 11.0.194)

However, when I ran the command "python run.py --config configs/nerf/hotdog.py --render_test", the overflow issue occurred.

Following is the Traceback code.

Traceback (most recent call last):
  File "run.py", line 630, in <module>
    train(args, cfg, data_dict)
  File "run.py", line 545, in train
    data_dict=data_dict, stage='coarse')
  File "run.py", line 449, in scene_rep_reconstruction
    **render_kwargs)
  File "/opt/pyenv/versions/mlhw/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/DirectVoxGO/lib/dvgo.py", line 309, in forward
    rays_o=rays_o, rays_d=rays_d, **render_kwargs)
  File "/home/user/DirectVoxGO/lib/dvgo.py", line 288, in sample_ray
    ray_pts, mask_outbbox, ray_id, step_id, N_steps, t_min, t_max = render_utils_cuda.sample_pts_on_rays(
RuntimeError: value cannot be converted to type int without overflow

I see the class TensoRFGrid in the grid.py, have you reproduce the TensoRF in your project?

Thanks for your great work!
I am reading your codes, and find that you init the DenseGrid with zero. Do we need to init DenseGrid with random noise? Just like how you did in TensoRFGrid.

DirectVoxGO/lib/grid.py

Line 45 in 341e1fc

self.grid = nn.Parameter(torch.zeros([1, channels, *world_size]))

It is always hard to config a matched version of pytorch/cuda /torch_scatter ENV for developers using other framework such as onnx and tensorflow, I've tried many times and still got stuck at torch_scatter 😴😴.

Could you kindly attach the .onnx format pre-trained model file? @sunset1995
many thanks!

Hi

congrats on your work ! I have been trying to use your code on objects with a stationary camera. This means that the camera stays still and I rotate the object. But I am getting really bad results. Do you have an ideia of why this is happening ?

Thanks in advance and again awesome work :D

Hi,
I would like to congratulate for this wonderful work!
Additionally I would also point out some discrepancies that I found between Train and Test results.
I'm training an inward facing real world scene: the scene is composed by 61 images covering redundantly the depicted object.
Generated images from training images camera poses have PSNR ≈ 26 while test images PSNR is ≈ 16.
Generated test images seem to be affected by big occluding clouds and distortions.
I'm using a slightly modified version of the ./configs/custom/default_ubd_inward_facing.py configuration file which I paste below.
Images are 1616x1080. Colmap's extimated camera poses and 3d point cloud seem to be accurate.

Can you suggest some parameters that I should tune differently in order to get better Test results?

base = '../default.py'

expname = 'scene7'
basedir = './data/scene7'

data = dict(
datadir='./data/scene7',
dataset_type='llff',
spherify=True,
factor=2,
llffhold=0,
bd_factor=None,
white_bkgd=True,
rand_bkgd=True,
unbounded_inward=True,
load2gpu_on_the_fly=True,
)

coarse_train = dict(N_iters=0)

fine_train = dict(
N_iters=80000,
N_rand=4096,
lrate_decay=80,
ray_sampler='flatten',
weight_nearclip=0.0,
weight_distortion=0.01,
pg_scale=[2000,4000,6000,8000,10000,12000,14000,16000],
tv_before=20000,
tv_dense_before=20000,
weight_tv_density=1e-6,
weight_tv_k0=1e-7,
)

alpha_init = 1e-4
stepsize = 0.5

fine_model_and_render = dict(
num_voxels=(3203)*2,
num_voxels_base=(3203)2,
alpha_init=alpha_init,
stepsize=stepsize,
fast_color_thres={
'delete': True,
0 : alpha_initstepsize/10,
1500: min(alpha_init, 1e-4)*stepsize/5,
2500: min(alpha_init, 1e-4)*stepsize/2,
3500: min(alpha_init, 1e-4)*stepsize/1.5,
4500: min(alpha_init, 1e-4)*stepsize,
5500: min(alpha_init, 1e-4),
6500: 1e-4,
},
world_bound_scale=1,
)

coarse_model_and_render = dict(
num_voxels=10240002, # expected number of voxel
num_voxels_base=10240002, # to rescale delta distance
density_type='DenseGrid', # DenseGrid, TensoRFGrid
k0_type='DenseGrid', # DenseGrid, TensoRFGrid
density_config=dict(),
k0_config=dict(),
mpi_depth=128, # the number of planes in Multiplane Image (work when ndc=True)
nearest=False, # nearest interpolation
pre_act_density=False, # pre-activated trilinear interpolation
in_act_density=False, # in-activated trilinear interpolation
bbox_thres=1e-3, # threshold to determine known free-space in the fine stage
mask_cache_thres=1e-3, # threshold to determine a tighten BBox in the fine stage
rgbnet_dim=0, # feature voxel grid dim
rgbnet_full_implicit=False, # let the colors MLP ignore feature voxel grid
rgbnet_direct=True, # set to False to treat the first 3 dim of feature voxel grid as diffuse rgb
rgbnet_depth=6, # depth of the colors MLP (there are rgbnet_depth-1 intermediate features)
rgbnet_width=128, # width of the colors MLP
alpha_init=1e-6, # set the alpha values everywhere at the begin of training
fast_color_thres=1e-7, # threshold of alpha value to skip the fine stage sampled point
maskout_near_cam_vox=True, # maskout grid points that between cameras and their near planes
world_bound_scale=1, # rescale the BBox enclosing the scene
stepsize=0.3, # sampling stepsize in volume rendering
)

I prepare my data in nsvf data, the train is noraml, but when render new view the result is vertical flip，so I try to set inverse_y = false, but the train result is bad, can you tell me how to prepare own data in nsvf format?

Hi, thanks for your great work. The code that calculates the alpha value from density uses this one :

alpha = 1 - (1 + exp(density + shift)) ^ (-interval)

However, I found that the original nerf uses the equation to calculate alpha from density as :

alpha = 1 - exp(-density * interval)

My question is

(1) What is the difference between those two representations?

(2) Why do we have to use a custom cuda kernel for calculating alpha instead of using torch library? I am curious about this in terms of the improvement of speed.

llff_config set ndc=True, why llff type data type ndc=True, but other datatype set ndc=False

Hi, really great and interesting work!!!!! Congratulations!!!!!

I have a question about the work, hope you can help me. How could I extract the object mesh from the learned radiance fields models?

Best regards,

Hi, thanks for your wonderful work!
I'm working on reproducing your results on Madoka and Otobai, after training via your given configs. I can only get results with low PSNRs (~31 for Madoka and ~24 for Otobai). What could be the problems there?

Hello,
after installing the requirements file (note: I suggest you explicitly add older versions of torch and torchvision to the requirements file, since the newest versions aren't compatible with torch-scatter) and taking care of those problems, I attempt to run the code:
python run.py --config configs/tankstemple/Barn.py

but I get the following error:
Traceback (most recent call last):
File "run.py", line 5, in
import mmcv
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/init.py", line 4, in
from .fileio import *
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/fileio/init.py", line 2, in
from .file_client import BaseStorageBackend, FileClient
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/fileio/file_client.py", line 15, in
from mmcv.utils.misc import has_method
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/utils/init.py", line 39, in
from .env import collect_env
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/utils/env.py", line 13, in
from .parrots_wrapper import get_build_config
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/utils/parrots_wrapper.py", line 94, in
BuildExtension, CppExtension, CUDAExtension = _get_extension()
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/mmcv/utils/parrots_wrapper.py", line 64, in _get_extension
from torch.utils.cpp_extension import (BuildExtension, CppExtension,
File "/cs/labs/peleg/yairbleiberg/directvoxgovenv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 23, in
from pkg_resources import packaging # type: ignore[attr-defined]
ImportError: cannot import name 'packaging' from 'pkg_resources' (unknown location)

I looked online, but wasn't able to find any help. any idea what's going on?

Thanks :)

Hi @sunset1995. Great work and impressive results. I'm a PhD student at University of Toronto, and a researcher on a commercial company. I'm interested in trying your ideas for some potential commercial product in the company. I mainly rely on your efficient CUDA implementation part.

I checked your GPL license, which says only research usage is allowed. And the license expert says I can't add this cuda code to our codebase under current license.

If you choose GPL because of part of the code is forked from a GPL-licensed repo, my presumptuous request is that if you'll consider redistributing the cuda part in a more permissive license (e.g. gist under MIT). Otherwise, feel free to reject my request and close the issue.

Best wishes.

I want to save the voxel grid as a file.

support these features?

Hi,

I have trained an inward-facing scene of a can using 120 posed images from the YCB dataset in an equally spaced upper semi-sphere and used the 'nerf' config e.g.

_base_ = '../default.py'

expname = 'dvgo_Masterchefcan'
basedir = './logs/ycbrealworld'

data = dict(
    datadir='./data/ycbrealworld/002_master_chef_can',
    dataset_type='ycbrealworld',
    white_bkgd=True,
)

coarse_train = dict(
    pervoxel_lr_downrate=2,
    N_iters=10000,
)

fine_train = dict(
    N_iters=10000
)

Training images

Test renderings

Rendering the training poses works well, but the test poses have lots of distortion:

Coarse trained model

Viewing the coarse model, it looks like it is dense in a semi-spherical area around the can (essentially where all the cameras are pointing):

Is this normal? Is it likely to be the problem? How can this be fixed?

Thanks!

I capture some pictures using my phone, but when i finish the colmap mapper process. it send out this message. I find code related to it.
pts_arr = [] vis_arr = [] for k in pts3d: pts_arr.append(pts3d[k].xyz) cams = [0] * poses.shape[-1] for ind in pts3d[k].image_ids: if len(cams) < ind - 1: print('ERROR: the correct camera poses for current points cannot be accessed') return cams[ind-1] = 1 vis_arr.append(cams)
But i have no idea. Could you help me?

Dear authors,

Many thanks for open sourcing the impressive work. I got nice results (PSNR nearly 26) when training on my custom data with the given parameters in [configs/custom/Madoka.py], but I'm having trouble with TensoRFGrid (PSNR nearly 18). I appended the following congfigs according to [configs/nerf/ship.tensorf.py] for training with TensoRFGrid.

fine_model_and_render = dict(
    num_voxels=384**3,
    density_type='TensoRFGrid',
    density_config=dict(n_comp=8),
    k0_type='TensoRFGrid',
    k0_config=dict(n_comp=24),
)

I have tried doubling num_voxels or n_comp, and it didn't work. Do you have any ideas? Hope to receive your reply.

Thanks for the amazing work. I tried to run the scrips on nerf_llff dataset using the provided llff config file but I can't seem to reproduce the results. The average psnr I got was ~17 which is quite a lot lower than the reported results in the dvgo v2 paper.
Could you please suggest changes in configurations that I can make to get better results?

I tried to reconstruct a human in the middle using the cmu panoptic data, but there seems to be several reconstruced human in the scene, how can I adjust the camera system?

Hi,

Thanks for sharing your work, I am getting some trouble executing your script with success using nerf_synthetic dataset as described in README, process seems to be killed because there is not enough memory, but not sure if it's VRAM or RAM.
Similar output with Evulation or video rendering commands.
Here is the output. Do you have any idea how to reduce memory consumption ?
Thanks

Using /home/x0s/.cache/torch_extensions/py310_cu102 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/x0s/.cache/torch_extensions/py310_cu102/adam_upd_cuda/build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module adam_upd_cuda...
Using /home/x0s/.cache/torch_extensions/py310_cu102 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/x0s/.cache/torch_extensions/py310_cu102/render_utils_cuda/build.ninja...
Building extension module render_utils_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module render_utils_cuda...
Using /home/x0s/.cache/torch_extensions/py310_cu102 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/x0s/.cache/torch_extensions/py310_cu102/total_variation_cuda/build.ninja...
Building extension module total_variation_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module total_variation_cuda...
Using /home/x0s/.cache/torch_extensions/py310_cu102 as PyTorch extensions root...
No modifications detected for re-loaded extension module render_utils_cuda, skipping build step...
Loading extension module render_utils_cuda...
Using /home/x0s/.cache/torch_extensions/py310_cu102 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/x0s/.cache/torch_extensions/py310_cu102/ub360_utils_cuda/build.ninja...
Building extension module ub360_utils_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module ub360_utils_cuda...
Loaded blender (400, 800, 800, 4) torch.Size([160, 4, 4]) [800, 800, 1111.1110311937682] ./data/nerf_synthetic/lego
Killed

Hi,

I've got a question regarding the CUDA extensions, more precise the following function:

How difficult would it be to add a gradient for this function so that pytorch's autograd can backpropagate trough?

Hi @sunset1995, thank you for this work! The code is very clear and useful.

I am trying to train the model on a scene (scene0000_00) from ScanNet dataset. I got the model to train correctly, but the rendering results are slightly poor, for example (top left: ground truth, top right: rendered image, bottom right: rendered depth):

The train PSNR reaches a high value of 25, but test PSNR is only 13. Also, the rendered images and depths look pretty good on the training images, but fail on testset image as shown image.

I am wondering what can be done to improve the performance. I tried Instant-NGP with similar near_far parameters (i.e. [0.1, 10.0]) and the results were decent. Can you please advise?

Hi, thank you for providing the new version. However, when I first test the code, I meet the follow error. Do you have suggestions on solving the problem?
My environment is
torch 1.8.1 cuda10.2 python3.7.4

python run.py --config configs/nerf/chair.py --render_test

Using /home/hl589/.cache/torch_extensions as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/hl589/.cache/torch_extensions/adam_upd_cuda/build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] :/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output adam_upd_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include/TH -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include/THC -isystem :/usr/local/cuda/include -isystem /home/hl589/.conda/envs/Directvoxgo_new/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -std=c++14 -c /home/hl589/DirectVoxGO_new/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o
FAILED: adam_upd_kernel.cuda.o
:/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output adam_upd_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include/TH -isystem /home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/include/THC -isystem :/usr/local/cuda/include -isystem /home/hl589/.conda/envs/Directvoxgo_new/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -std=c++14 -c /home/hl589/DirectVoxGO_new/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o
/bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "run.py", line 13, in
from lib import utils, dvgo, dmpigo
File "/home/hl589/DirectVoxGO_new/lib/utils.py", line 11, in
from .masked_adam import MaskedAdam
File "/home/hl589/DirectVoxGO_new/lib/masked_adam.py", line 10, in
verbose=True)
File "/home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in _jit_compile
is_standalone=is_standalone)
File "/home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1407, in _write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/home/hl589/.conda/envs/Directvoxgo_new/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'adam_upd_cuda'

First of all, thank you for the awesome work!

I think it would be useful to log the losses and the metric in tensorboard. Is there any plan to add this feature?

Thanks for the amazing work. I run the scrip likes "readme.md" says. But the error appear at render_utils_cuda.sample_pts_on_rays. Here is the error info:

Using /home/chambin/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/chambin/.cache/torch_extensions/py38_cu113/adam_upd_cuda/build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module adam_upd_cuda...
Using /home/chambin/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/chambin/.cache/torch_extensions/py38_cu113/render_utils_cuda/build.ninja...
Building extension module render_utils_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module render_utils_cuda...
Using /home/chambin/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/chambin/.cache/torch_extensions/py38_cu113/total_variation_cuda/build.ninja...
Building extension module total_variation_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module total_variation_cuda...
Using /home/chambin/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
No modifications detected for re-loaded extension module render_utils_cuda, skipping build step...
Loading extension module render_utils_cuda...
Using /home/chambin/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/chambin/.cache/torch_extensions/py38_cu113/ub360_utils_cuda/build.ninja...
Building extension module ub360_utils_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module ub360_utils_cuda...
Loaded blender (120, 200, 200, 4) torch.Size([160, 4, 4]) [200, 200, 274.7477419454622] ./data/nerf_synthetic/plane_grass
train: start
compute_bbox_by_cam_frustrm: start
/home/chambin/anaconda3/envs/dvgo2/lib/python3.8/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755903507/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
compute_bbox_by_cam_frustrm: xyz_min tensor([-3.0437, -3.0213, -2.6208])
compute_bbox_by_cam_frustrm: xyz_max tensor([3.0345, 3.0083, 2.3107])
compute_bbox_by_cam_frustrm: finish
scene_rep_reconstruction (coarse): train from scratch
scene_rep_reconstruction (coarse): use dense voxel grid
dvgo: set density bias shift to tensor([-13.8155], device='cpu')
dvgo: voxel_size tensor(0.0561)
dvgo: world_size tensor([108, 107, 87])
dvgo: voxel_size_base tensor(0.0561)
dvgo: voxel_size_ratio tensor(1.)
create_optimizer_or_freeze_model: param density lr 0.1
create_optimizer_or_freeze_model: param k0 lr 0.1
create_optimizer_or_freeze_model: param rgbnet not exist
get_training_rays: start
get_training_rays: finish (eps time: 0.024027109146118164 sec)
dvgo: voxel_count_views start
dvgo: voxel_count_views finish (eps time: 1.5023679733276367 sec)
0%| | 0/5000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "run.py", line 630, in
train(args, cfg, data_dict)
File "run.py", line 541, in train
scene_rep_reconstruction(
File "run.py", line 446, in scene_rep_reconstruction
render_result = model(
File "/home/chambin/anaconda3/envs/dvgo2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/chambin/Documents/projects/DirectVoxGO-main/lib/dvgo.py", line 310, in forward
ray_pts, ray_id, step_id = self.sample_ray(
File "/home/chambin/Documents/projects/DirectVoxGO-main/lib/dvgo.py", line 290, in sample_ray
ray_pts, mask_outbbox, ray_id, step_id, N_steps, t_min, t_max = render_utils_cuda.sample_pts_on_rays(
RuntimeError: value cannot be converted to type int without overflow
`

My environment info is:
ubuntu 21 cuda 11.3 torch 1.11

Could anyone tell me how to fix this error? Thanks in advance!

hi, thanks for your work.

I am using CUDA 11.3, Pytorch 1.11.0. When I run python run.py --config configs/lf/ship.py --render_test, I get:

Using /home/s/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/s/.cache/torch_extensions/py39_cu113/adam_upd_cuda/build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module adam_upd_cuda...
Using /home/s/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/s/.cache/torch_extensions/py39_cu113/render_utils_cuda/build.ninja...
Building extension module render_utils_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module render_utils_cuda...
Using /home/s/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/s/.cache/torch_extensions/py39_cu113/total_variation_cuda/build.ninja...
Building extension module total_variation_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module total_variation_cuda...
Using /home/s/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
No modifications detected for re-loaded extension module render_utils_cuda, skipping build step...
Loading extension module render_utils_cuda...
Using /home/s/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Creating extension directory /home/s/.cache/torch_extensions/py39_cu113/ub360_utils_cuda...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/s/.cache/torch_extensions/py39_cu113/ub360_utils_cuda/build.ninja...
Building extension module ub360_utils_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF ub360_utils.o.d -DTORCH_EXTENSION_NAME=ub360_utils_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include/TH -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda-11.3/include -isystem /home/s/anaconda3/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp -o ub360_utils.o
In file included from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/c10/core/DeviceType.h:8,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/c10/core/Device.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/c10/core/Allocator.h:6,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/ATen/ATen.h:7,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/extension.h:4,
from /home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp:1:
/home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp: In function ‘at::Tensor cumdist_thres(at::Tensor, float)’:
/home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp:11:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
11 | #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
| ^
/home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp:13:24: note: in expansion of macro ‘CHECK_CUDA’
13 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
| ^~~~~~~~~~
/home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp:16:3: note: in expansion of macro ‘CHECK_INPUT’
16 | CHECK_INPUT(dist);
| ^~~~~~~~~~~
In file included from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/ATen/core/Tensor.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/ATen/DeviceGuard.h:4,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/ATen/ATen.h:11,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/extension.h:4,
from /home/s/DirectVoxGO/lib/cuda/ub360_utils.cpp:1:
/home/s/anaconda3/lib/python3.9/site-packages/torch/include/ATen/core/TensorBody.h:210:30: note: declared here
210 | DeprecatedTypeProperties & type() const {
| ^~~~
[2/3] /usr/local/cuda-11.3/bin/nvcc -DTORCH_EXTENSION_NAME=ub360_utils_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include/TH -isystem /home/s/anaconda3/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda-11.3/include -isystem /home/s/anaconda3/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++14 -c /home/s/DirectVoxGO/lib/cuda/ub360_utils_kernel.cu -o ub360_utils_kernel.cuda.o
[3/3] c++ ub360_utils.o ub360_utils_kernel.cuda.o -shared -L/home/s/anaconda3/lib/python3.9/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/usr/local/cuda-11.3/lib64 -lcudart -o ub360_utils_cuda.so
Loading extension module ub360_utils_cuda...
Traceback (most recent call last):
File "/home/s/DirectVoxGO/run.py", line 593, in
data_dict = load_everything(args=args, cfg=cfg)
File "/home/s/DirectVoxGO/run.py", line 167, in load_everything
data_dict = load_data(cfg.data)
File "/home/s/DirectVoxGO/lib/load_data.py", line 127, in load_data
images, poses, render_poses, hwf, K, i_split = load_nerfpp_data(args.datadir)
File "/home/s/DirectVoxGO/lib/load_nerfpp.py", line 122, in load_nerfpp_data
K_flatten = np.loadtxt(tr_K[0])
IndexError: list index out of range

I am not solving the above error. How can I solve this? I hope to hear from you soon!! Thank you:)

Hello, with the recent updates there is a missing requirement pytorch_scatter.

I don't know what is the best to go for putting it in requirements.txt as the version depends on PyTorch version and CUDA version, see https://github.com/rusty1s/pytorch_scatter#pytorch-1100 , so I let you find a way.

Typically, the format is pip install torch-scatter -f https://data.pyg.org/whl/torch-${TORCH_VERSION}+${CUDA_VERSION}.html where ${TORCH_VERSION}+${CUDA_VERSION} is e.g. 1.10.0+cu111.

Thank you!

Thanks for sharing this amazing work !👍👍

My team is following your work and I need to do a comparative experiment between DVGOv1 and DVGOv2, so can you release or share DVGOv1 source code (w/o cuda and distortion loss)?

Thank for your time and looking forward to your future work!

Dear authors,

Many thanks for open sourcing the impressive work. I came to notice that in order to calculate the depth maps, in

Best regards.

Hello everyone, thank you for releasing this great technique. I was testing you technoque on you processed tenaks and Temples scenes, however, I noticed that the Ignatius scene seems to be having a problem with the Intrinsics. It is formated in a diferent way than the other scenes and the folder structure is also different. Because of this, I could not test in this scene. Thank you for the attention!

Hello authors,

Thank you for your great work and your code. I am trying to run the model, which involves building the CUDA extension. I am aware that issue #13 exists, but it does not provide information that solves my issue, so I am opening a new issue.

I am using CUDA 11.6 with PyTorch built for CUDA 11.6. I have successfully installed pytorch-scatter and all the dependencies in requirements.txt. When I run python run.py --config configs/nerf/lego.py --render_test, I get:

>>> python run.py --config configs/nerf/lego.py --render_test

Using /tmp/torch-ext as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /tmp/torch-ext/adam_upd_cuda/build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] /usr/local/cuda/bin/nvcc  -ccbin /software/compilers/gcc-5.4.0/bin/gcc -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSI
ON_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /users/user/miniconda
3/envs/new2/lib/python3.9/site-packages/torch/include -isystem /users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/t
orch/csrc/api/include -isystem /users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/TH -isystem /users/user/minicon
da3/envs/new2/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /users/user/miniconda3/envs/new2/incl
ude/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__C
UDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-opt
ions '-fPIC' -std=c++14 -c /users/user/projects/experiments/active/DirectVoxGO/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o
FAILED: adam_upd_kernel.cuda.o
/usr/local/cuda/bin/nvcc  -ccbin /software/compilers/gcc-5.4.0/bin/gcc -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSION_H -
DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /users/user/miniconda3/envs
/new2/lib/python3.9/site-packages/torch/include -isystem /users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/c
src/api/include -isystem /users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/TH -isystem /users/user/miniconda3/en
vs/new2/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /users/user/miniconda3/envs/new2/include/py
thon3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO
_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '
-fPIC' -std=c++14 -c /users/user/projects/experiments/active/DirectVoxGO/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o
/users/user/projects/experiments/active/DirectVoxGO/lib/cuda/adam_upd_kernel.cu: In lambda function:
/users/user/projects/experiments/active/DirectVoxGO/lib/cuda/adam_upd_kernel.cu:74:116: warning: ‘T* at::Tensor::data() const [with T = dou
ble]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/ATen/core/TensorBody.h:236:1: note: declared here
   T * data() const {
 ^
/users/user/projects/experiments/active/DirectVoxGO/lib/cuda/adam_upd_kernel.cu:74:141: warning: ‘T* at::Tensor::data() const [with T = dou
ble]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/ATen/core/TensorBody.h:236:1: note: declared here
   T * data() const {

...

 ^
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h: In instantiation of
 ‘std::shared_ptr<torch::nn::Module> torch::nn::Cloneable<Derived>::clone(const c10::optional<c10::Device>&) const [with Derived = torch::nn:
:CrossMapLRN2dImpl]’:
/tmp/tmpxft_0000447a_00000000-6_adam_upd_kernel.cudafe1.stub.c:59:27:   required from here
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:58:59: error: invali
d static_cast from type ‘const torch::OrderedDict<std::basic_string<char>, at::Tensor>’ to type ‘torch::OrderedDict<std::basic_string<char>,
at::Tensor>&’
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:71:61: error: invali
d static_cast from type ‘const torch::OrderedDict<std::basic_string<char>, std::shared_ptr<torch::nn::Module> >’ to type ‘torch::OrderedDict<
std::basic_string<char>, std::shared_ptr<torch::nn::Module> >&’
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h: In instantiation of
 ‘std::shared_ptr<torch::nn::Module> torch::nn::Cloneable<Derived>::clone(const c10::optional<c10::Device>&) const [with Derived = torch::nn:
:EmbeddingBagImpl]’:

...

/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:58:59: error: invali
d static_cast from type ‘const torch::OrderedDict<std::basic_string<char>, at::Tensor>’ to type ‘torch::OrderedDict<std::basic_string<char>,
at::Tensor>&’
/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:71:61: error: invali
d static_cast from type ‘const torch::OrderedDict<std::basic_string<char>, std::shared_ptr<torch::nn::Module> >’ to type ‘torch::OrderedDict<
std::basic_string<char>, std::shared_ptr<torch::nn::Module> >&’
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1865, in _run_ninja_build
    subprocess.run(
  File "/users/user/miniconda3/envs/new2/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/users/user/projects/experiments/active/DirectVoxGO/run.py", line 13, in <module>
    from lib import utils, dvgo, dcvgo, dmpigo
  File "/users/user/projects/experiments/active/DirectVoxGO/lib/utils.py", line 11, in <module>
    from .masked_adam import MaskedAdam
  File "/users/user/projects/experiments/active/DirectVoxGO/lib/masked_adam.py", line 7, in <module>
    adam_upd_cuda = load(
  File "/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1257, in load
    return _jit_compile(
  File "/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1480, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1594, in _write_ninja_file_and_bui
ld_library
    _run_ninja_build(
  File "/users/user/miniconda3/envs/new2/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1881, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'adam_upd_cuda'

Your assistance on this issue would be greatly appreciated! Thank you again.

I have not found any description of maskout_lt_nview, would you please give me some advice?

Thanks for your great work. I want to know where you set "world_size" for density constuction? And what's the shape of it?
https://github.com/sunset1995/DirectVoxGO/blob/main/lib/dvgo.py#L48