nvlabs / nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".

nvdiffrec

Joint optimization of topology, materials and lighting from multi-view image observations as described in the paper Extracting Triangular 3D Models, Materials, and Lighting From Images.

For differentiable marching tetrahedons, we have adapted code from NVIDIA's Kaolin: A Pytorch Library for Accelerating 3D Deep Learning Research.

News

• 2023-10-20 : We added a version of the renderutils library written in slangpy to leverage the autodiff capabilities of slang instead of CUDA extensions with manually crafted forward and backward passes. This simplifies the code substantially, with the same runtime performance as before. This version is available in the slang branch of this repo.

• 2023-09-15 : We added support for the FlexiCubes isosurfacing technique. Please see the config configs/bob_flexi.json for a usage example, and refer to the FlexiCubes documentation for details.

Citation

@inproceedings{Munkberg_2022_CVPR,
    author    = {Munkberg, Jacob and Hasselgren, Jon and Shen, Tianchang and Gao, Jun and Chen, Wenzheng 
                    and Evans, Alex and M\"uller, Thomas and Fidler, Sanja},
    title     = "{Extracting Triangular 3D Models, Materials, and Lighting From Images}",
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {8280-8290}
}

Licenses

Copyright © 2022, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License.

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing.

Installation

Requires Python 3.6+, VS2019+, Cuda 11.3+ and PyTorch 1.10+

Tested in Anaconda3 with Python 3.9 and PyTorch 1.10

One time setup (Windows)

Install the Cuda toolkit (required to build the PyTorch extensions). We support Cuda 11.3 and above. Pick the appropriate version of PyTorch compatible with the installed Cuda toolkit. Below is an example with Cuda 11.6

conda create -n dmodel python=3.9
activate dmodel
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
pip install ninja imageio PyOpenGL glfw xatlas gdown
pip install git+https://github.com/NVlabs/nvdiffrast/
pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn#subdirectory=bindings/torch
imageio_download_bin freeimage

Every new command prompt

activate dmodel

Examples

Our approach is designed for high-end NVIDIA GPUs with large amounts of memory. To run on mid-range GPU's, reduce the batch size parameter in the .json files.

Simple genus 1 reconstruction example:

python train.py --config configs/bob.json

Visualize training progress (only supported on Windows):

python train.py --config configs/bob.json --display-interval 20

Multi GPU example (Linux only. Experimental: all results in the paper were generated using a single GPU), using PyTorch DDP

torchrun --nproc_per_node=4 train.py --config configs/bob.json

Below, we show the starting point and the final result. References to the right.

The results will be stored in the out folder. The Spot and Bob models were created and released into the public domain by Keenan Crane.

Included examples

• spot.json - Extracting a 3D model of the spot model. Geometry, materials, and lighting from image observations.
• spot_fixlight.json - Same as above but assuming known environment lighting.
• spot_metal.json - Example of joint learning of materials and high frequency environment lighting to showcase split-sum.
• bob.json - Simple example of a genus 1 model.

Datasets

We additionally include configs (nerf_*.json, nerd_*.json) to reproduce the main results of the paper. We rely on third party datasets, which are courtesy of their respective authors. Please note that individual licenses apply to each dataset. To automatically download and pre-process all datasets, run the download_datasets.py script:

activate dmodel
cd data
python download_datasets.py

Below follows more information and instructions on how to manually install the datasets (in case the automated script fails).

NeRF synthetic dataset Our view interpolation results use the synthetic dataset from the original NeRF paper. To manually install it, download the NeRF synthetic dataset archive and unzip it into the nvdiffrec/data folder. This is required for running any of the nerf_*.json configs.

NeRD dataset We use datasets from the NeRD paper, which features real-world photogrammetry and inaccurate (manually annotated) segmentation masks. Clone the NeRD datasets using git and rescale them to 512 x 512 pixels resolution using the script scale_images.py. This is required for running any of the nerd_*.json configs.

activate dmodel
cd nvdiffrec/data/nerd
git clone https://github.com/vork/ethiopianHead.git
git clone https://github.com/vork/moldGoldCape.git
python scale_images.py

Server usage (through Docker)

• Build docker image.

cd docker
./make_image.sh nvdiffrec:v1

• Start an interactive docker container: docker run --gpus device=0 -it --rm -v /raid:/raid -it nvdiffrec:v1 bash

• Detached docker: docker run --gpus device=1 -d -v /raid:/raid -w=[path to the code] nvdiffrec:v1 python train.py --config configs/bob.json

nvdiffrec's Issues

Confusing mesh result on nerd_ehead

Thank you for the great work!
I followed the one time setup on ubuntu 18.04. Python=3.9, CUDA=11.3, PyTorch=1.10.
Then I had a trial on nerd_ehead by running:
python train.py --config configs/nerd_ehead.json
In dmtet_mesh folder, the mesh looks good :

But in mesh folder, the mesh.obj looks confusing:

I did not change any code in this repo so I did not know why this happened. Could you please help me solve this problem? Thank you for your time!

i found that the version of VS2019 may have to be lower than 16.11

i tried to install nvdiffrec on my computer and failed when the version of VS was 16.11, and it worked when i reinstall VS2019 16.9.4
the fail was like "ninja -v "and" missing a '(' "

[deleted]

train program is stucked

DatasetNERF: 100 images with shape [800, 800]
DatasetNERF: 200 images with shape [800, 800]
Encoder output: 32 dims
Warning: Lock file exists in build directory: 'C:\Users\AppData\Local\torch_extensions\torch_extensions\Cache\py39_cu113\renderutils_plugin\lock'
Using C:\Users\AppData\Local\torch_extensions\torch_extensions\Cache\py39_cu113 as PyTorch extensions root...

surface subdivision

Dear author,
Is surface subdivision module released in this code or somewhere else?

Thanks

Does this work with just a single image?

I am thinking of giving this a try, but for most of what I'd want to sketch out in 3d I'd only have a single ref image. Could this algorithm give me anything useful in such a situation or does it require shots from all sides? Of course I am not expecting a full 3d model that includes the sides missing in the image, but merely something that could be used a bit more flexibly compared to taking the same image and putting it on a plane.

improve the quality of generated mesh

Dear author,

Can you show us how to reproduce the fine meshes presented in the paper?

I have run on Lego scene, but the mesh is not smooth.

How to configurate everthing to get better mesh?

Thanks

Is there any way to use datasets from instant ngp?

Hi, I am trying to train nvfdiffrec straigth from instant-ngp NERF datasets but I'm blocked. The instant-ngp dataset uses images without alpha and it seems like nvdiffrec needs images with alpha. Also, nvdiffrec uses a train/test/val dataset, is there a way to use the same input as instant-ngp uses?

Thanks!

Pending after computing MSE and PSNR

Hello,
Recently i made myself test data to practically check what resolution and parameters is suitable for my desktop, i made the resolution of 1264x1264, and last it pended even for 24 hours, i checked it consumed the approximate 7408g GPU, so, is my data too rough(MSE - 0.2591xx, last img_loss=1.109134) or the other reason?

So, someone give me a share/suggestion about how to do to my high resolution please? e.g. the relationship, 11g GPU: max resolution and scale.

In addition: actually my images is very high resolution(3000x4000), but to my GPU(3080 11g), i scale that to 1264x1264 after some tries.

Hardware: DESKTOP RTX 3080(11g), cuda 11.3.
Software: ubuntu20.04
Running GPU cost during pending: always fixed cost, approximate 7.xg
Running output and strace log: see the following.

The console output:
python3 train.py --config ./configs/nerf_handong.json
Config / Flags:

config ./configs/nerf_handong.json
iter 2000
batch 1
spp 1
layers 4
train_res [1264, 1264]
display_res [1264, 1264]
texture_res [2048, 2048]
display_interval 0
save_interval 100
learning_rate [0.03, 0.01]
min_roughness 0.08
custom_mip False
random_textures True
background white
loss logl1
out_dir out/nerf_handong
ref_mesh data/nerf_synthetic/handong
base_mesh None
validate True
mtl_override None
dmtet_grid 128
mesh_scale 2.75
env_scale 1.0
envmap None
display [{'latlong': True}, {'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}]
camera_space_light False
lock_light False
lock_pos False
sdf_regularizer 0.2
laplace relative
laplace_scale 3000
pre_load True
kd_min [0.0, 0.0, 0.0, 0.0]
kd_max [1.0, 1.0, 1.0, 1.0]
ks_min [0, 0.1, 0.0]
ks_max [1.0, 1.0, 1.0]
nrm_min [-1.0, -1.0, 0.0]
nrm_max [1.0, 1.0, 1.0]
cam_near_far [0.1, 1000.0]
learn_light True
local_rank 0
multi_gpu False

DatasetNERF: 28 images with shape [1264, 1264]
DatasetNERF: 28 images with shape [1264, 1264]
Encoder output: 32 dims
Using /home/xx/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/xx/.cache/torch_extensions/py38_cu113/renderutils_plugin/build.ninja...
Building extension module renderutils_plugin...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module renderutils_plugin...
iter= 0, img_loss=1.166577, reg_loss=0.334082, lr=0.02999, time=468.3 ms, rem=15.61 m
...
iter= 1990, img_loss=0.936701, reg_loss=0.014961, lr=0.01199, time=484.8 ms, rem=4.85 s
iter= 2000, img_loss=1.109134, reg_loss=0.014806, lr=0.01194, time=503.9 ms, rem=0.00 s
Running validation
MSE, PSNR
0.25911513, 6.480

^C^CKilled

strace log:
sched_yield() = 0
sched_yield() = 0
sched_yield() = 0
...
sched_yield() = 0
sched_yield() = 0

Thanks for your contribution!

Regards

Does the GPU shared memory can be used inside this feature?

Hello,
Now i run this featuren under windows11, and i would like to see that the shared memory of the GPU can relex the memory issue, but, still it failed to run for the higher resolution images.

So i would like to know if the shared memory can be used inside this feature? (of course, maybe it needs tiny-cuda-nn to solve???).

Image resolution: 3648x3648.
GPU: RTX 3090 24g

{
"ref_mesh": "data/nerf_synthetic/xxxx",
"random_textures": true,
"iter": 5000,
"save_interval": 100,
"texture_res": [ 1024, 1024 ],
"train_res": [3648, 3648],
"batch": 1,
"learning_rate": [0.03, 0.01],
"ks_min" : [0, 0.1, 0.0],
"dmtet_grid" : 64,
"mesh_scale" : 1.5,
"laplace_scale" : 3000,
"display": [{"latlong" : true}, {"bsdf" : "kd"}, {"bsdf" : "ks"}, {"bsdf" : "normal"}],
"layers" : 4,
"background" : "white",
"out_dir": "nerf_xxxx"
}

Loading extension module renderutils_plugin...
Traceback (most recent call last):
File "D:\zhansheng\proj\windows\nvdiffrec\train.py", line 594, in
geometry, mat = optimize_mesh(glctx, geometry, mat, lgt, dataset_train, dataset_validate,
File "D:\zhansheng\proj\windows\nvdiffrec\train.py", line 415, in optimize_mesh
img_loss, reg_loss = trainer(target, it)
File "C:\Users\jinshui\anaconda3\envs\dmodel\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "D:\zhansheng\proj\windows\nvdiffrec\train.py", line 299, in forward
return self.geometry.tick(glctx, target, self.light, self.material, self.image_loss_fn, it)
File "D:\zhansheng\proj\windows\nvdiffrec\geometry\dmtet.py", line 218, in tick
buffers = self.render(glctx, target, lgt, opt_material)
File "D:\zhansheng\proj\windows\nvdiffrec\geometry\dmtet.py", line 209, in render
return render.render_mesh(glctx, opt_mesh, target['mvp'], target['campos'], lgt, target['resolution'], spp=target['spp'],
File "D:\zhansheng\proj\windows\nvdiffrec\render\render.py", line 231, in render_mesh
layers += [(render_layer(rast, db, mesh, view_pos, lgt, resolution, spp, msaa, bsdf), rast)]
File "D:\zhansheng\proj\windows\nvdiffrec\render\render.py", line 166, in render_layer
buffers = shade(gb_pos, gb_geometric_normal, gb_normal, gb_tangent, gb_texc, gb_texc_deriv,
File "D:\zhansheng\proj\windows\nvdiffrec\render\render.py", line 46, in shade
all_tex = material['kd_ks_normal'].sample(gb_pos)
File "D:\zhansheng\proj\windows\nvdiffrec\render\mlptexture.py", line 90, in sample
p_enc = self.encoder(_texc.contiguous())
File "C:\Users\jinshui\anaconda3\envs\dmodel\lib\site-packages\torch\nn\modules\module.py", line 1128, in _call_impl
result = forward_call(*input, **kwargs)
File "C:\Users\jinshui\anaconda3\envs\dmodel\lib\site-packages\tinycudann\modules.py", line 119, in forward
output = _module_function.apply(
File "C:\Users\jinshui\anaconda3\envs\dmodel\lib\site-packages\tinycudann\modules.py", line 31, in forward
native_ctx, output = native_tcnn_module.fwd(input, params)
RuntimeError: C:/Users/jinshui/AppData/Local/Temp/pip-req-build-ii64hvij/include\tiny-cuda-nn/gpu_memory.h:558 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: C:/Users/jinshui/AppData/Local/Temp/pip-req-build-ii64hvij/include\tiny-cuda-nn/gpu_memory.h:462 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE

Regards

How to run under linux ?

Can you give a detailed instruction about how to run under linux ?

Use implicit model

Hi

the mesh seems be explicitly learned, how to modify the config for implicit learning?

Thanks

Underlying buffer has been detached

I followed

conda create -n dmodel python=3.9
activate dmodel
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install ninja imageio PyOpenGL glfw xatlas gdown
pip install git+https://github.com/NVlabs/nvdiffrast/
pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
imageio_download_bin freeimage

in Windows 10.

pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

threw

                instantiation of "decltype(auto) std::_Get_unwrapped(_Iter &&) [with _Iter=nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>> *const &]"
    e:/VS/VC/Tools/MSVC/14.29.30133/include\xmemory(1703): here
                instantiation of "std::_Alloc_ptr_t<_Alloc> std::_Uninitialized_move(_InIt, _InIt, std::_Alloc_ptr_t<_Alloc>, _Alloc &) [with _InIt=nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>> *, _Alloc=std::_Rebind_alloc_t<std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>, nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>]"
    e:/VS/VC/Tools/MSVC/14.29.30133/include\vector(1651): here
                instantiation of "void std::vector<_Ty, _Alloc>::_Umove_if_noexcept1(std::vector<_Ty, _Alloc>::pointer, std::vector<_Ty, _Alloc>::pointer, std::vector<_Ty, _Alloc>::pointer, std::true_type) [with _Ty=nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, _Alloc=std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>]"
    e:/VS/VC/Tools/MSVC/14.29.30133/include\vector(1662): here
                instantiation of "void std::vector<_Ty, _Alloc>::_Umove_if_noexcept(std::vector<_Ty, _Alloc>::pointer, std::vector<_Ty, _Alloc>::pointer, std::vector<_Ty, _Alloc>::pointer) [with _Ty=nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, _Alloc=std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>]"
    e:/VS/VC/Tools/MSVC/14.29.30133/include\vector(1297): here
                instantiation of "void std::vector<_Ty, _Alloc>::_Reallocate_exactly(std::vector<_Ty, _Alloc>::size_type) [with _Ty=nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, _Alloc=std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>]"
    e:/VS/VC/Tools/MSVC/14.29.30133/include\vector(1363): here
                instantiation of "void std::vector<_Ty, _Alloc>::reserve(std::vector<_Ty, _Alloc>::size_type) [with _Ty=nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, _Alloc=std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(18616): here
                instantiation of "void nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::json_value::destroy(nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::value_t) [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(19828): here
                instantiation of "nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::~basic_json() [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(20679): here

...

    e:/VS/VC/Tools/MSVC/14.29.30133/include\xutility(124): error: expected a "("
              detected during:
                instantiation of "void *std::_Voidify_iter(_Iter) [with _Iter=std::vector<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>> *]"
    e:/VS/VC/Tools/MSVC/14.29.30133/include\xmemory(681): here
                instantiation of "void std::_Default_allocator_traits<_Alloc>::construct(_Alloc &, _Objty *, _Types &&...) [with _Alloc=std::allocator<std::vector<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>>>, _Objty=std::vector<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>>, _Types=<>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(18440): here
                instantiation of "T *nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::create<T,Args...>(Args &&...) [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>, T=std::vector<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>, std::allocator<nlohmann::basic_json<std::map, std::vector, std::string, __nv_bool, int64_t, uint64_t, double, std::allocator, nlohmann::adl_serializer, std::vector<uint8_t, std::allocator<uint8_t>>>>>, Args=<>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(18517): here
                instantiation of "nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::json_value::json_value(nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::value_t) [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(18947): here
                instantiation of "nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::basic_json(nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::value_t) [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(18971): here
                instantiation of "nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::basic_json(std::nullptr_t) [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(24402): here
                instantiation of "nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType> nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::parse(IteratorType, IteratorType, nlohmann::basic_json<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFloatType, AllocatorType, JSONSerializer, BinaryType>::parser_callback_t, __nv_bool, __nv_bool) [with ObjectType=std::map, ArrayType=std::vector, StringType=std::string, BooleanType=__nv_bool, NumberIntegerType=int64_t, NumberUnsignedType=uint64_t, NumberFloatType=double, AllocatorType=std::allocator, JSONSerializer=nlohmann::adl_serializer, BinaryType=std::vector<uint8_t, std::allocator<uint8_t>>, IteratorType=const char *]"
    E:/Temp/pip-req-build-dx4hpd_b/dependencies\json/json.hpp(26513): here

    Error limit reached.
    100 errors detected in the compilation of "E:/Temp/pip-req-build-dx4hpd_b/src/cpp_api.cu".
    Compilation terminated.
    cpp_api.cu
    [5/5] cl /showIncludes /nologo /O2 /W3 /GL /DNDEBUG /MD /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -IE:\Temp\pip-req-build-dx4hpd_b/include -IE:\Temp\pip-req-build-dx4hpd_b/dependencies -IE:\Temp\pip-req-build-dx4hpd_b/dependencies/cutlass/include -IE:\Temp\pip-req-build-dx4hpd_b/dependencies/cutlass/tools/util/include -IE:\miniconda\envs\dmodel\lib\site-packages\torch\include -IE:\miniconda\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IE:\miniconda\envs\dmodel\lib\site-packages\torch\include\TH -IE:\miniconda\envs\dmodel\lib\site-packages\torch\include\THC "-IE:\Eigene Programme\Cuda\include" -IE:\miniconda\envs\dmodel\include -IE:\miniconda\envs\dmodel\Include "-IE:\VS\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IE:\VS\VC\Tools\MSVC\14.29.30133\include" "-IE:\WK\NETFXSDK\4.8\include\um" "-IE:\Windows Kits\10\include\10.0.19041.0\ucrt" "-IE:\Windows Kits\10\include\10.0.19041.0\shared" "-IE:\Windows Kits\10\include\10.0.19041.0\um" "-IE:\Windows Kits\10\include\10.0.19041.0\winrt" "-IE:\Windows Kits\10\include\10.0.19041.0\cppwinrt" -c E:\Temp\pip-req-build-dx4hpd_b\bindings\torch\tinycudann\bindings.cpp /FoE:\Temp\pip-req-build-dx4hpd_b\bindings\torch\build\temp.win-amd64-3.9\Release\tinycudann/bindings.obj /std:c++14 -DTCNN_MIN_GPU_ARCH=52 -DTCNN_NO_NETWORKS -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
   
   ...
    Hinweis: Einlesen der Datei: E:\Temp\pip-req-build-dx4hpd_b/dependencies\json/json.hpp
    Hinweis: Einlesen der Datei:  E:\VS\VC\Tools\MSVC\14.29.30133\include\cassert
    Hinweis: Einlesen der Datei:   E:\Windows Kits\10\include\10.0.19041.0\ucrt\assert.h
    Hinweis: Einlesen der Datei: E:\Temp\pip-req-build-dx4hpd_b/dependencies\pybind11_json/pybind11_json.hpp
    Hinweis: Einlesen der Datei: E:\Temp\pip-req-build-dx4hpd_b/include\tiny-cuda-nn/cpp_api.h
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "E:\miniconda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1740, in _run_ninja_build
        subprocess.run(
      File "E:\miniconda\envs\dmodel\lib\subprocess.py", line 528, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "E:\Temp\pip-req-build-dx4hpd_b\bindings/torch\setup.py", line 117, in <module>
        setup(
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\core.py", line 148, in setup
        return run_commands(dist)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\core.py", line 163, in run_commands
        dist.run_commands()
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\dist.py", line 1214, in run_command
        super().run_command(command)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
        cmd_obj.run()
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\command\install.py", line 68, in run
        return orig.install.run(self)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\install.py", line 664, in run
        self.run_command('build')
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\dist.py", line 1214, in run_command
        super().run_command(command)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
        cmd_obj.run()
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build.py", line 135, in run
        self.run_command(cmd_name)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\dist.py", line 1214, in run_command
        super().run_command(command)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
        cmd_obj.run()
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\command\build_ext.py", line 79, in run
        _build_ext.run(self)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 339, in run
        self.build_extensions()
      File "E:\miniconda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 741, in build_extensions
        build_ext.build_extensions(self)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension
        _build_ext.build_extension(self, ext)
      File "E:\miniconda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 528, in build_extension
        objects = self.compiler.compile(sources,
      File "E:\miniconda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 714, in win_wrap_ninja_compile
        _write_ninja_file_and_compile_objects(
      File "E:\miniconda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1419, in _write_ninja_file_and_compile_objects
        _run_ninja_build(
      File "E:\miniconda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1756, in _run_ninja_build
        raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
    Error in atexit._run_exitfuncs:
    Traceback (most recent call last):
      File "E:\miniconda\envs\dmodel\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
        return stream.closed
    ValueError: underlying buffer has been detached
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'E:\miniconda\envs\dmodel\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Temp\\pip-req-build-dx4hpd_b\\bindings/torch\\setup.py'"'"'; __file__='"'"'E:\\Temp\\pip-req-build-dx4hpd_b\\bindings/torch\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-networks install --record 'E:\Temp\pip-record-xxbieno2\install-record.txt' --single-version-externally-managed --compile --install-headers 'E:\miniconda\envs\dmodel\Include\tinycudann' Check the logs for full command output.

docker running problem

hello, why I use docker to run train.py and the trianing process only runs 1 round, while I use colab to run , it runs 2 rounds, to be more specific, the colab running result contains 2 parts, dmtet_mesh and mesh, but there's only dmtet_mesh when using docker. sorry i currently haven't look all through your code, and I wonder what caused this problem. thank you for your time.

how to perform training with our own dataset ? Will it produce same results with drone captured images ?

caffe2.proto module import error

After installing, executing train.py like in the ReadMe causes the following

ImportError: cannot import name 'metanet_pb2' from partially initialized module 'caffe2.proto' (most likely due to a circular import) (C:\Users\John\miniconda3\lib\site-packages\caffe2\proto\__init__.py)

Looking at it in VSCode it looks like caffe2 isn't installed? Does it need to be built from source seperately?

Also everywhere it looks like caffe2 is deprecated?

Recreating MLP parametrized SDF experiments

Hello,

First of all, I would like to thank you for sharing this great work.
I'm trying to run some experiments on a custom scene I shot and it looks like I'm getting the high-frequency noise you mentioned in the paper (re. DTU dataset), so I wanted to test the MLP parameterization method you suggested, to see if it improves the results.
As it's not implemented in the code, I tried adding a NeRF-like MLP to predict SDF values (both with frequency encodings and hash-grid), but I'm having trouble initializing the weights and training with it.
Can you give some more information on how to initialize the training in this case, or if I should use a different loss or hyper parameters for t_f, n_base, etc..

The core dumping occurs when training with fixed topology in pass 2.

Thanks a lot for your contribution for the community.

I use the docker environment and the gpu is V-100.

Runing the script, train.py, and there is an error that "Floating point exception (core dumped” when training with fixed topology in pass 2.

build tiny-cuda-nn error

Hello there, I have a problem compiling the tiny-cuda-nn. I compiled it on a win10 system with RTX3090, CUDA11.3 and VS2019. Somehow I just cannot compile the tiny-cuda-nn and I got too many errors that seem to come from VS. Can you help me out?

Render the final result?

Hello. First of all, thank you very much for your excellent work.
I'm sorry I'm not very familiar with graphics engines, but now I want to render the final output, such as hotdog's Mesh. There are some questions below hope to get your answer:
1.According to the description in the article, kd.png\ks.png\kn.png describes the material of the model，But the final output has.mtL file, is.mtL equivalent to three PNG files？Now when I use mesh.load_mesh(mesh.obj), the api will directly use mesh.mtl instead of loading three Png files
2. If.mtL means the same as three Png files, I can render by loading the model and loading Png files?

Models turn to swiss cheese after >5000 iters

What loss is supposed to control regularity? I've been focusing on this relatively simple model to see what sort of values works well in training for handheld video, and at least the lefthand outer image looks like it fits the images until 5k iters (1 batchsize) into training, the actual model looks very irregular, and finally collapses near the end of training tests.

the colmap (both exhaustive and sequential) tracking look like they map accurately, the paper describes a loss to solve this issue, but I don't know if the total loss must be reduced due to batchsize, or the loss for model regularity must be increased (which I dont know the config line for).
30% of the dataset images have been manually removed due to motion blur and being too far off the edges, which have helped in early training, and slight reduction in early collapses.

I have included the dataset and the key iters that start collapsing (0-1000 iters, 4000-6000 iters) and the final models in the zip below (updated with more compressed iter images):
https://files.catbox.moe/3v79c6.zip

Is the training scheme running through the dataset sequentially, and therefore the final iters failing due to the images at the end? Both passes seem to fail at near end of the session. If so, then randomizing the images (if captured from video) would spread the bad frames out, instead of destroying the model at the end of training.

Unable to run reproduce example, due to type error (unexpected indexing mode).

I am having trouble following the instructions from the readme file. Whenever I run the training for an example, I receive an error:

Traceback (most recent call last):
  File "C:\Users\Admin\source\repos\nvdiffrec\test\train.py", line 563, in <module>
    dataset_train    = DatasetMesh(ref_mesh, glctx, RADIUS, FLAGS, validate=False)
  File "C:\Users\Admin\source\repos\nvdiffrec\test\dataset\dataset_mesh.py", line 46, in __init__
    self.envlight = light.load_env(FLAGS.envmap, scale=FLAGS.env_scale)
  File "C:\Users\Admin\source\repos\nvdiffrec\test\render\light.py", line 141, in load_env
    return _load_env_hdr(fn, scale)
  File "C:\Users\Admin\source\repos\nvdiffrec\test\render\light.py", line 132, in _load_env_hdr
    cubemap = util.latlong_to_cubemap(latlong_img, [512, 512])
  File "C:\Users\Admin\source\repos\nvdiffrec\test\render\util.py", line 106, in latlong_to_cubemap
    gy, gx = torch.meshgrid(torch.linspace(-1.0 + 1.0 / res[0], 1.0 - 1.0 / res[0], res[0], device='cuda'),
TypeError: meshgrid() got an unexpected keyword argument 'indexing'

I am not sure if this is related, but I also received some warnings during installation, namely when attempting to install tinycudann:

Running setup.py install for tinycudann ...
    WARNING: Subprocess output does not appear to be encoded as cp1252

I then receive similar encoding warning every time I run the training script:

C:\Users\Admin\AppData\Roaming\Python39\site-packages\torch\utils\cpp_extension.py:304:
    UserWarning: Error checking compiler version for cl: 'utf-8' codec can't decode byte 0x81 in position 62: invalid start byte

Just in case:

• I have CUDA 11.5 installed and first tried to manually install appropriate cuda toolkit using conda install cudatoolkit=11.5 -c conda-forge after which conda install pytorch torchvision torchaudio cudatoolkit=11.5 -c pytorch also succeeded, but none of this solved the issue.
• GPU is RTX 2080 Ti
• Windows 10 x64

Weirdly enough, removing the indexing parameter from all calls to torch.meshgrid() appears to fix the issue. The parameter defaults to ij anyway, but this would only work for current releases of pytorch, as the documentation states, that it will transition to xy as the default in the future. No idea why it works when removing the argument entirely, though.

I would be grateful for any hints. Thanks! 🙂

Error in installation

run this line:
pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

then get this error :
ERROR: Command errored out with exit status 1:
command: 'C:\Users\ReZaN.conda\envs\dmodel\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] ="'"'C:\Users\ReZaN\AppData\Local\Temp\pip-req-build-9qo96lf9\bindings/torch\setup.py'"'"'; file='"'"'C:\Users\ReZaN\AppData\Local\Temp\pip-req-build-9qo96lf9\bindings/torch\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\ReZaN\AppData\Local\Temp\pip-pip-egg-info-gryf3c1b'
cwd: C:\Users\ReZaN\AppData\Local\Temp\pip-req-build-9qo96lf9\bindings/torch
Complete output (6 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ReZaN\AppData\Local\Temp\pip-req-build-9qo96lf9\bindings/torch\setup.py", line 41, in
raise RuntimeError("Could not locate a supported Microsoft Visual C++ installation")
RuntimeError: Could not locate a supported Microsoft Visual C++ installation
Building PyTorch extension for tiny-cuda-nn version 1.6
----------------------------------------
WARNING: Discarding git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

How to show mesh with textures?

Thank you for the amazing work! Now, I have trained the Nerf datasets of chair and I get a great result. I want to show the mesh with texture in MeshLab, but the chair mesh has nontexture in MeshLab. So what method should I use?

Trouble installing tinycudann

Whenever i try running this command pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch it spits out ERROR: Command errored out with exit status 1

I tried building it manually this way

but it just fails to build it then spits out the same error.
Tried the command below and get RuntimeError: Error compiling objects for extension
fully_fused_mlp.cu.txt
I also tried to cmake it, no luck

Cuda: 11.3
Python 3.9
Windows 10
CMake 3.23.2
Visual Studio 2019 community
Git 2.36.1

Mesh surface roughness obtained using nerd data

Thank you for your amazing work. I made a data set in nerd format. After training, the surface of mesh is not very good. What parameters should I modify?

My config is like this：
{
"ref_mesh": "data/nerd/shoe",
"random_textures": true,
"iter": 20000,
"save_interval": 100,
"texture_res": [ 2048, 2048 ],
"train_res": [960, 540],
"batch": 4,
"learning_rate": [0.03, 0.03],
"kd_min" : [0.03, 0.03, 0.03],
"kd_max" : [0.8, 0.8, 0.8],
"ks_min" : [0, 0.08, 0],
"ks_max" : [0, 1.0, 1.0],
"dmtet_grid" : 128,
"mesh_scale" : 3,
"camera_space_light" : true,
"background" : "white",
"display" : [{"bsdf":"kd"}, {"bsdf":"ks"}, {"bsdf" : "normal"}],
"out_dir": "shoe"
}

About how to generate NERF validate database and if it have some rule to make test data as validation.

Hello,
Now, i want to make my training data to generate 3d model, but i saw it have three kinds of database(train/test/val), and the val database maybe not used(???), so who can help share the way to how to generate validate_database(corresponding to the data folder test) and what rule to make it as TEST data? because as what i understand the test database should be the accurate data from the implementation.
In addition, i see this kind of file(r_0_depth_0000.png) are not used inside implementation, right?

 What i know:
 a. i know the train database(data folder 'data') and **transforms_train.json** can generate by **COLMAP,** but don't know what rule to generate TEST data.


So,  someone can share the information about my issue please!

Regards

error in one time setup

when I run: pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
i get

    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\nunob\anaconda3\envs\dmodel\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\nunob\\AppData\\Local\\Temp\\pip-req-build-ajsps1vv\\bindings/torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\nunob\\AppData\\Local\\Temp\\pip-req-build-ajsps1vv\\bindings/torch\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\nunob\AppData\Local\Temp\pip-pip-egg-info-gm8vy1dn'
         cwd: C:\Users\nunob\AppData\Local\Temp\pip-req-build-ajsps1vv\bindings/torch
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\nunob\AppData\Local\Temp\pip-req-build-ajsps1vv\bindings/torch\setup.py", line 3, in <module>
        import torch
      File "C:\Users\nunob\anaconda3\envs\dmodel\lib\site-packages\torch\__init__.py", line 126, in <module>
        raise err
    OSError: [WinError 182] The operating system cannot run %1. Error loading "C:\Users\nunob\anaconda3\envs\dmodel\lib\site-packages\torch\lib\shm.dll" or one of its dependencies.

How to run inference using custom images?

I'm really sorry if this has been asked before.

1. Do I understand correctly that to run this you need a pretrained model or does it generate a 3D mesh from the training?
2. If it does need a pretrained model, is there a download link somewhere and example code on how to run inference on our own set of custom images?

I want to use this as a method to replace photogrammetry. So taking 100 pictures, or however many is needed of a real physical object, and running them through this model.

Error when running sample spot.json

Has anyone else gotten a RuntimeError when attempting to run the sample code given in the setup?

I get:

RuntimeError: Error building extension 'nvdiffrast_plugin': [1/1] "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64/link.exe" common.o glutil.o rasterize.cuda.o rasterize.o int
erpolate.cuda.o texture.cuda.o texture.o antialias.cuda.o torch_bindings.o torch_rasterize.o torch_interpolate.o torch_texture.o torch_antialias.o /nologo /DLL /LIBPATH:D:\Download\nvdiffrec\venv\lib\site-packages\nvdiffrast\torch\.
.\lib /DEFAULTLIB:gdi32 /DEFAULTLIB:opengl32 /DEFAULTLIB:user32 /DEFAULTLIB:setgpu c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?_torch_cuda_cu_linker_symbol_op_cuda@native@at@@YA?AVTensor@2@AEBV32@@Z torch_cuda_cpp
.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:D:\Download\nvdiffrec\venv\lib\site-packages\torch\lib torch_python.lib /LIBPATH:D:\Download\nvdiffrec\venv\Scripts\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Tool
kit\CUDA\v11.7\lib/x64" cudart.lib /out:nvdiffrast_plugin.pyd
FAILED: nvdiffrast_plugin.pyd
"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64/link.exe" common.o glutil.o rasterize.cuda.o rasterize.o interpolate.cuda.o texture.cuda.o texture.o antialias.cuda.o torch_bi
ndings.o torch_rasterize.o torch_interpolate.o torch_texture.o torch_antialias.o /nologo /DLL /LIBPATH:D:\Download\nvdiffrec\venv\lib\site-packages\nvdiffrast\torch\..\lib /DEFAULTLIB:gdi32 /DEFAULTLIB:opengl32 /DEFAULTLIB:user32 /D
EFAULTLIB:setgpu c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?_torch_cuda_cu_linker_symbol_op_cuda@native@at@@YA?AVTensor@2@AEBV32@@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:D:\Down
load\nvdiffrec\venv\lib\site-packages\torch\lib torch_python.lib /LIBPATH:D:\Download\nvdiffrec\venv\Scripts\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\lib/x64" cudart.lib /out:nvdiffrast_plugin.pyd
LINK : fatal error LNK1104: cannot open file 'python39.lib'
ninja: build stopped: subcommand failed.

A full printout of the error is attached:
nvdiffrec-error-printout.txt

Custom dataset config options

I've been able to run this on my own data (slowly, on a 3070) using U^2Net matting and colmap2nerf code from instant-ngp (if others are following this, remove the image extensions, and it should run on windows)
The issue I'm having is val, and test data or the lack of it. Are there config options to remove the requirement for those datasets. Copying and renaming the json lists to val and train works, but is rather cumbersome, and I was wondering if I was missing a better option with the NeRD dataset preparation, which use manual masks not applied to the image itself (which is possible with U^2 net, and would help with colmap mapping since the background can still be used for feature tracking.)

I haven't looked into what data is required to be presented in the config, nor resolution/training options, but I'm just wondering what's the generally intended arguments and method for presenting your own data, when no val is available.

Would it be possible to modify this to also use CPU memory?

I looked at issue #2 and I'm curious if you could modify the project to use CPU memory by swapping in and out images and other data structures as needed? (Not sure if https://github.com/IBM/pytorch-large-model-support could be used for the tensor part). I noticed the limitations section in the paper mentioned the memory consumption already, so if this isn't easy then feel free to close this.

Part of this is I'm interested in seeing what happens with larger and more images. (This assumes training duration isn't a high priority and model quality is the goal. I have a 3090 and a lot of DDR5 memory for reference).

How to get a model with higer accuracy and more details?

Hello,

I have run "python train.py --config configs/nerf_lego.json" with default setting. The picture below shows my mesh output.

1. The smooth surface is not flat (as shown in the red box). How should I improve it?
2. Lots of details are loss due to the geometric simplification. How can I get a model with more details?

Thank you in advance!

Low Error - But Low Quality Tet Mesh

I've run a few tests with some in-the-wild examples from my photogrammetry collection, and I seem to be consistently running into the same issue: the combined output converges to something that is very, very similar to the target images, but the quality of actual tet mesh is garbage compared to what you get from the nerd data.

For instance, from 19 images:

The rendered output clearly converges pretty close to the desired result and kd seems maybe close-ish to what you'd want, but the probe, ks and n don't converge at all on what you would expect as correct values. The resulting tet mesh holds roughly the right shape, but generally just looks like swiss cheese:

Another example from 45 images:

This one generated a better mesh, but seemed to over-optimise ks for a surface which should largely be very diffuse. On the inside, the mesh is still very much swiss cheese.

This seems in contrast to very solid results on synthetic data. Do you have any recommendations for how to tune the training for better results? I have used similar configs to what are used for the nerd datasets, but I'm not sure if there are better ways of controlling the learning rates. Or is the trick ultimately that it just needs more photos?

Edit: Happy to share and open-source these datasets if that would be at all helpful for future testing.

except blender

Dear author,

Except blender, are there other renderers that can render the output in interactive rate?

Thanks

is there a code demo?

is there a code demo?

CO3D dataset

Hi. Thanks for sharing the code. Great work!

I am trying to train the model with objects from the CO3D dataset (link). However, I am getting pretty bad results and i suspect that my camera matrices are incorrect. I would really appriciate some help if possible.

Here is how i create the matrices (i used this article as a reference to create the projection matrix) :

def get_camera(self, annotations):
        image_size = annotations["image"]["size"]
        viewpoint_frame = annotations["viewpoint_frame"]
        R = viewpoint_frame['R']
        T = viewpoint_frame['T']
        focal_length = viewpoint_frame['focal_length']
        principal_point = torch.tensor(viewpoint_frame['principal_point'])

        mv = torch.tensor([
            [R[0][0],R[0][1],R[0][2], T[0]],
            [R[1][0],R[1][1],R[1][2], T[1]],
            [R[2][0],R[2][1],R[2][2], T[2]],
            [    0 ,     0,      0,    1  ],
        ], dtype=torch.float32)

        # Transform mv from Pytorch3D to OpenGL coordinates 
        rotate_y = util.rotate_y(math.pi)
        mv = rotate_y @ mv

        # Convert principal_point and focal_length from NDC space to pixel space
        half_image_size = torch.tensor(list(reversed(image_size))) / 2.0
        principal_point_px = (-1.0 * (principal_point-1) * half_image_size)
        focal_length_px =  torch.tensor(focal_length) * half_image_size

        A = 2*focal_length_px[0] / image_size[1] #2*f/w
        B = 2*focal_length_px[1] / image_size[0] #2*f/h
        C = (image_size[1] - 2*principal_point_px[0]) / image_size[1] #(w – 2*cx)/w
        D = (image_size[0] - 2*principal_point_px[1]) / image_size[0] #(h – 2*cy)/h
        n=0.1
        f=1000.0

        proj = torch.tensor([
            [A, 0, -C, 0],
            [0, -B, D, 0],
            [0, 0.0, (-f - n) / (f - n), -2.0*f*n/(f-n)],
            [0, 0, -1, 0],
        ])

        campos = torch.linalg.inv(mv)[:3, 3]
        mvp    = proj @ mv

Result:

OSError: PyTorch CUDA is unavailable. tinycudann requires PyTorch to be installed with the CUDA backend.

I am trying to install this on windows(Radeon RX 570, if that matters)

    ERROR: Command errored out with exit status 1:
     command: 'C:\ProgramData\Anaconda3\envs\dmodel\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Sammy_HD\\AppData\\Local\\Temp\\pip-req-build-heecsmjx\\bindings/torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\Sammy_HD\\AppData\\Local\\Temp\\pip-req-build-heecsmjx\\bindings/torch\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\Sammy_HD\AppData\Local\Temp\pip-pip-egg-info-4is6wjv_'
         cwd: C:\Users\Sammy_HD\AppData\Local\Temp\pip-req-build-heecsmjx\bindings/torch
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Sammy_HD\AppData\Local\Temp\pip-req-build-heecsmjx\bindings/torch\setup.py", line 115, in <module>
        raise EnvironmentError("PyTorch CUDA is unavailable. tinycudann requires PyTorch to be installed with the CUDA backend.")
    OSError: PyTorch CUDA is unavailable. tinycudann requires PyTorch to be installed with the CUDA backend.
    Building PyTorch extension for tiny-cuda-nn version 1.6
    ----------------------------------------

The command I was able to isolate was

pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Any GPU recommendation

First, fantastic work.

Compute time on my 3080 is slow and i have many cuda out of memory errors.

Do you have any GPU recommendation here?
Would a RTX A6000 help with the out of memory issues and fasten the process?

The rendered images have some white blocks.

when I fix the mesh(without any material) to learn the material, the rendered images have some white blocks.

Error when doing first time setup on Windows

I get the following error when running the command pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch imageio_download_bin freeimage on Windows 10:

    100 errors detected in the compilation of "C:/Users/Graham/AppData/Local/Temp/pip-req-build-od5yzghh/src/encoding.cu".
    Compilation terminated.
    encoding.cu
    [5/5] cl /showIncludes /nologo /O2 /W3 /GL /DNDEBUG /MD /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -IC:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh/include -IC:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh/dependencies -IC:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh/dependencies/cutlass/include -IC:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh/dependencies/cutlass/tools/util/include -IC:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\include -IC:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\include\TH -IC:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\Users\Graham\.conda\envs\dmodel\include -IC:\Users\Graham\.conda\envs\dmodel\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -c C:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh\bindings\torch\tinycudann\bindings.cpp /FoC:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh\bindings\torch\build\temp.win-amd64-3.9\Release\tinycudann/bindings.obj /std:c++14 -DTCNN_MIN_GPU_ARCH=75 -DTCNN_NO_NETWORKS -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1717, in _run_ninja_build
        subprocess.run(
      File "C:\Users\Graham\.conda\envs\dmodel\lib\subprocess.py", line 528, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Graham\AppData\Local\Temp\pip-req-build-od5yzghh\bindings/torch\setup.py", line 117, in <module>
        setup(
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\core.py", line 148, in setup
        return run_commands(dist)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\core.py", line 163, in run_commands
        dist.run_commands()
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\dist.py", line 1214, in run_command
        super().run_command(command)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
        cmd_obj.run()
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\command\install.py", line 68, in run
        return orig.install.run(self)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\install.py", line 664, in run
        self.run_command('build')
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\dist.py", line 1214, in run_command
        super().run_command(command)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
        cmd_obj.run()
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build.py", line 135, in run
        self.run_command(cmd_name)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\dist.py", line 1214, in run_command
        super().run_command(command)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
        cmd_obj.run()
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\command\build_ext.py", line 79, in run
        _build_ext.run(self)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 339, in run
        self.build_extensions()
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 735, in build_extensions
        build_ext.build_extensions(self)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension
        _build_ext.build_extension(self, ext)
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 528, in build_extension
        objects = self.compiler.compile(sources,
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 708, in win_wrap_ninja_compile
        _write_ninja_file_and_compile_objects(
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1399, in _write_ninja_file_and_compile_objects
        _run_ninja_build(
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1733, in _run_ninja_build
        raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
    Error in atexit._run_exitfuncs:
    Traceback (most recent call last):
      File "C:\Users\Graham\.conda\envs\dmodel\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
        return stream.closed
    ValueError: underlying buffer has been detached
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\Graham\.conda\envs\dmodel\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Graham\\AppData\\Local\\Temp\\pip-req-build-od5yzghh\\bindings/torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\Graham\\AppData\\Local\\Temp\\pip-req-build-od5yzghh\\bindings/torch\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-networks install --record 'C:\Users\Graham\AppData\Local\Temp\pip-record-2fzq7zhs\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\Graham\.conda\envs\dmodel\Include\tinycudann' Check the logs for full command output.```

Windows: Windows 10
Cuda: 11.3
PyTorch: 1.10
Visual Studio: Visual Studio 2019 with build tools installed

train.py not found

After setting up with everything under Installation and doing all the One time setup commands after trying to do:
python train.py --config configs/bob.json
or
python train.py --config configs/bob.json --display-interval 20
I get this:
(dmodel) C:\WINDOWS\system32>python train.py --config configs/bob.json --display-interval 20
python: can't open file 'C:\WINDOWS\system32\train.py': [Errno 2] No such file or directory

I am sorry this might be a dumb question but I am stumped on how to fix it.

MLP texture does not support for DLMesh

Thank you for the amazing work!!

I tried to enable the mlp texture for DLM - changing this line https://github.com/NVlabs/nvdiffrec/blob/main/train.py#L632 to

mat = initial_guess_material(geometry, True, FLAGS)

However, when I run the the job, it produces the following error:

iter=    0, img_loss=0.020718, reg_loss=0.000000, lr=0.00010, time=606.2 ms, rem=10.10 m
Traceback (most recent call last):
  File "/home/wangjk/programs/nvdiffrec/train.py", line 778, in <module>
    geometry, mat = optimize_mesh(glctx, geometry, mat, lgt, dataset_train, dataset_validate, FLAGS, pass_idx=0, pass_name="mesh_pass", 
  File "/home/wangjk/programs/nvdiffrec/train.py", line 542, in optimize_mesh
    total_loss.backward()
  File "/home/wangjk/anaconda3/envs/torch-ngp/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/wangjk/anaconda3/envs/torch-ngp/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

Do you have any ideas about this? Really appreciate it ;)

Render with blender

thanks for your wonderful work! I try to render the output with blender, but i can't get the result the mess_pass output.

Abort during Rnning validation

Hello,
I got error at Running validation phase, the following is the log, thank you someone for your help!

i have changed the 'bach' to 1 to avoid memory error,

configs/nerf_chair.json:
{
"ref_mesh": "data/nerf_synthetic/chair",
"random_textures": true,
"iter": 5000,
"save_interval": 100,
"texture_res": [ 2048, 2048 ],
"train_res": [800, 800],
"batch": 1,
"learning_rate": [0.03, 0.01],
"ks_min" : [0, 0.08, 0.0],
"dmtet_grid" : 128,
"mesh_scale" : 2.1,
"laplace_scale" : 3000,
"display": [{"latlong" : true}, {"bsdf" : "kd"}, {"bsdf" : "ks"}, {"bsdf" : "normal"}],
"background" : "white",
"out_dir": "nerf_chair"
}

Running command: python3 train.py --config configs/nerf_chair.jsonf

Hardware:
"Quadro RTX 3000" : 6G.
System: UBUNTU20.04 + 64G

Error log:
Running validation
MSE, PSNR
0.00113444, 29.651
Traceback (most recent call last):
File "train.py", line 594, in
base_mesh = xatlas_uvmap(glctx, geometry, mat, FLAGS)
File "/home/jinshui/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(args, kwargs)
File "train.py", line 115, in xatlas_uvmap
mask, kd, ks, normal = render.render_uv(glctx, new_mesh, FLAGS.texture_res, eval_mesh.material['kd_ks_normal'])
File "/home/jinshui/workshop/prj/3d/nvdiffrec/render/render.py", line 266, in render_uv
rast, _ = dr.rasterize(ctx, uv_clip4, mesh.t_tex_idx.int(), resolution)
File "/home/jinshui/.local/lib/python3.8/site-packages/nvdiffrast-0.2.7-py3.8.egg/nvdiffrast/torch/ops.py", line 250, in rasterize
return _rasterize_func.apply(glctx, pos, tri, resolution, ranges, grad_db, -1)
File "/home/jinshui/.local/lib/python3.8/site-packages/nvdiffrast-0.2.7-py3.8.egg/nvdiffrast/torch/ops.py", line 184, in forward
out, out_db = _get_plugin().rasterize_fwd(glctx.cpp_wrapper, pos, tri, resolution, ranges, peeling_idx)
RuntimeError: Cuda error: 801[cudaGraphicsGLRegisterImage(&s.cudaColorBuffer[i], s.glColorBuffer[i], GL_TEXTURE_3D, cudaGraphicsRegisterFlagsReadOnly);]
terminate called after throwing an instance of 'c10::Error'
what(): Cuda error: 709[cudaGraphicsUnregisterResource(s.cudaColorBuffer[i]);]
Exception raised from rasterizeReleaseBuffers at /home/jinshui/.local/lib/python3.8/site-packages/nvdiffrast-0.2.7-py3.8.egg/nvdiffrast/common/rasterize.cpp:581 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f45a986f7d2 in /home/jinshui/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::string const&) + 0x5b (0x7f45a986be6b in /home/jinshui/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: rasterizeReleaseBuffers(int, RasterizeGLState&) + 0x25e (0x7f44d8e610e9 in /home/jinshui/.cache/torch_extensions/py38_cu113/nvdiffrast_plugin/nvdiffrast_plugin.so)
frame #3: RasterizeGLStateWrapper::~RasterizeGLStateWrapper() + 0x37 (0x7f44d8ebcf85 in /home/jinshui/.cache/torch_extensions/py38_cu113/nvdiffrast_plugin/nvdiffrast_plugin.so)
frame #4: std::default_delete::operator()(RasterizeGLStateWrapper) const + 0x26 (0x7f44d8ea29a6 in /home/jinshui/.cache/torch_extensions/py38_cu113/nvdiffrast_plugin/nvdiffrast_plugin.so)
frame #5: std::unique_ptr<RasterizeGLStateWrapper, std::default_delete >::~unique_ptr() + 0x56 (0x7f44d8e97392 in /home/jinshui/.cache/torch_extensions/py38_cu113/nvdiffrast_plugin/nvdiffrast_plugin.so)
frame #6: + 0xc4b2c (0x7f44d8e90b2c in /home/jinshui/.cache/torch_extensions/py38_cu113/nvdiffrast_plugin/nvdiffrast_plugin.so)
frame #7: + 0x1f5b20 (0x7f45b871ab20 in /home/jinshui/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #8: + 0x1f6cce (0x7f45b871bcce in /home/jinshui/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #9: python3() [0x5d1ea8]
frame #10: python3() [0x5a958d]

frame #12: python3() [0x6aa1ba]
frame #13: python3() [0x4ef8d8]
frame #19: __libc_start_main + 0xf3 (0x7f45bfa250b3 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

Regards

noise in rendered results (noise in diffuse/specular?)

Thank you for the amazing work!!

I did a quick trial on the custom in-the-wild data and found that there are some noise in fitting results. Did I mess up something? Have you observed this phonenmon before? Note that I fixed the vertex for mesh and just optimized the lighting / material.

rendered	reference

The texture maps looks like:

kd	ks	normal

Looking forward to your great help! Thanks a lot

can u please make that training visualation also works on linux

Visualize training progress (only supported on Windows)

Decoupling problem

First of all, thank you for your excellent work

Recently I have been doing some work on my own data set based on your project and I have noticed that Mesh reconstruction and rendering are very good.

But when I wanted to edit the light, I noticed that it had a strong connection to the material. Specifically, I can't get different lighting effects by directly replacing HDR files when rendering. But I noticed that in your article, the reconstructed hot dog was subjected to different light experiments, such as sunrise, evening and so on.

I wonder if you can only debug some simple lighting effects in Blender and not use different HDR files for rendering in the project, In addition, if possible, could you provide HDR files under different lighting conditions mentioned in the article?

Thanks

Implementations of MLPTexture3D

Hello,

Thanks for releasing the code for this amazing work!

I've been going through the code and find some implementations in TextureMLP3D confusing:

1. What's the meaning of scaling the gradient? I set gradient_scaling=1.0 and did not find much difference in the output.
2. Why not use the tcnn MLP but only the tcnn Encoding?

Thanks!

Created Colab for those with nb without gpus. Enjoy ;D

https://github.com/neurall/nvdiffrecColab/