nerfstudio-project / nerfstudio Goto Github PK
View Code? Open in Web Editor NEWA collaboration friendly studio for NeRFs
Home Page: https://docs.nerf.studio
License: Apache License 2.0
A collaboration friendly studio for NeRFs
Home Page: https://docs.nerf.studio
License: Apache License 2.0
This is a major refactoring, but we should change "Graph" to a name that is less confusing for first-time users attempting to understand our code. "Graph" has been too often confused with "computation graph".
We want to refactor how computing losses and metrics work. Currently get_loss_dict
returns a dictionary of losses. These losses are then combined in get_aggregated_loss_dict
using coefficients defined in the config. This workflow is not the most transparent, ie. if I add a new loss, I then need to know that I must update the configs accordingly.
Proposal:
Change get_loss_dict(outputs, batch)
to get_loss(outputs, batch, metrics=None, coefficients=None) -> float
and get_metrics_dict(outputs, batch) -> dict
.
Remove get_aggregated_loss_dict
Relevant code (would need to update for all models):
https://github.com/plenoptix/pyrad/blob/56661b5d9aa8adfec9cad60bce53036cb0ceca43/pyrad/graphs/vanilla_nerf.py#L143-L148
ImageDataset classes will return batches of images and masks, etc.
PixelSamplers will choose which pixels to use from the image datasets.
When running python scripts/run_train.py
on vanilla NeRF the following error is raised. The gather indices are out of bounds.
File "/projects/pyrad/pyrad/engine/trainer.py", line 205, in test_image
outputs = self.graph.get_outputs_for_camera_ray_bundle(camera_ray_bundle)
File "/home/tancik/miniconda3/envs/pyrad/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/projects/pyrad/pyrad/graphs/base.py", line 188, in get_outputs_for_camera_ray_bundle
outputs = self.forward_after_ray_generator(ray_bundle)
File "/projects/pyrad/pyrad/graphs/base.py", line 145, in forward_after_ray_generator
outputs = self.get_outputs(intersected_ray_bundle)
File "/projects/pyrad/pyrad/graphs/vanilla_nerf.py", line 121, in get_outputs
ray_samples_pdf = self.sampler_pdf(ray_bundle, ray_samples_uniform, weights_coarse)
File "/home/tancik/miniconda3/envs/pyrad/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/projects/pyrad/pyrad/graphs/modules/ray_sampler.py", line 47, in forward
ray_samples = self.generate_ray_samples(*args, **kwargs)
File "/home/tancik/miniconda3/envs/pyrad/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/projects/pyrad/pyrad/graphs/modules/ray_sampler.py", line 324, in generate_ray_samples
cdf_g1 = torch.gather(cdf, -1, above)
RuntimeError: CUDA error: device-side assert triggered
This is likely caused by #116
@liruilong940607
Rather than setting in the config, the user should be able to set the minimum and maximum render resolution from the viewer ui.
Allow user to pause training for smoother rendering.
The initialized b_matrix
at this line wouldn't be saved together with the model if it is not a buffer or parameter. You might want to consider adding self.register_buffer
to it.
I think we want these values to be the same in the config.
Can we determine if a config isn't used?
Perhaps an issue the recent changes that allow you to switch outputs?
Traceback (most recent call last):
File "scripts/run_train.py", line 221, in main
launch(
File "scripts/run_train.py", line 161, in launch
main_func(local_rank=0, world_size=1, config=config)
File "scripts/run_train.py", line 128, in _train
trainer.train()
File "/projects/pyrad/pyrad/engine/trainer.py", line 118, in train
self.visualizer_state.update_scene(step, self.graph)
File "/projects/pyrad/pyrad/viewer/server/viewer_utils.py", line 92, in update_scene
self._render_image_in_viewer(graph)
File "/projects/pyrad/pyrad/utils/profiler.py", line 34, in wrapper
ret = func(*args, **kwargs)
File "/projects/pyrad/pyrad/viewer/server/viewer_utils.py", line 226, in _render_image_in_viewer
image_output = outputs[output_type].cpu().numpy() * 255
KeyError: 'rgb'
create run_eval_dataset.py
File currently contains unused code and does not match the style guidelines.
I added some func to handle the colormap stuff
https://github.com/plenoptix/pyrad/blob/069cf2c40fb3ab68c483501f18713992b3c00d8a/pyrad/graphs/instant_ngp.py#L148-L162
And set it as a base class thing to get everything to work:
https://github.com/plenoptix/pyrad/blob/069cf2c40fb3ab68c483501f18713992b3c00d8a/pyrad/graphs/base.py#L139-L145
But i dont think this is best way to handle, so need to figure out a more robust way of handling this across implementations
This is where the function is referenced in visualizer code:
https://github.com/plenoptix/pyrad/blob/069cf2c40fb3ab68c483501f18713992b3c00d8a/pyrad/viewer/server/viewer_utils.py#L79
Issue of the F.grid_sample: padding_mode==zeros
means padding zero voxel values outside the grid. a query point that is slightly outside the grid would be interpolated by the voxel values on the boarder of the grid and the outside zero voxels. So it gives non-zero value at regions slightly outside the grid.
Toy code to show:
grid = torch.ones((1, 1, 128, 128, 128))
positions = torch.tensor([[0.5, 1.004, 0.5]])
values = F.grid_sample(
grid,
positions.view(1, -1, 1, 1, 3),
align_corners=True,
padding_mode="zeros",
)
print(values.flatten()) # >> 0.7460
Relevant code in our code base:
Currently configs need to specify train and test dataset configurations separately:
Intended behaviour should be the train dataset config is the default for the test dataset.
Only if test dataset is specified, then it will overwrite for test.
this is a strange location to put it...
Update the ipynb camera visualization with the following:
We should outline the steps to setup development environment and how to run the code checks. Will be useful/necessary when others want to contribute.
All of the components needed should already be implemented. Just need to create a graph/config and benchmark against the paper.
The goals of this PR are the following:
dataset_format
. Rather, it should know classes are implemented (see above) and choose appropriately.Bonus
Will involve linting
Make consistent with rest of code.
Pushing to git will now fail if there are warnings in the doc compilation. Can we add a doc compilation step in our run_action.py script so that we can catch those warning before pushing. This likely involves calling make clean
and make html
in the docs folder.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.