jonbarron / camp_zipnerf Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Hi,
Thank you for the open-sourced code.
I am digging the code to understand how to render an image.
However, when I checked dataset.camtoworlds matrix, the translations of z-axis are always 0. Below image shows a piece of the matrix value
I expect it should be x, y , and z values
Can anyone helps me to understand this.
Thank you
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.
Hi,
Really nice work on pose optimization. However, in order to evaluate the final rendering quality, I believe there will be a test-time optimization carried after training. I wonder if you can provide the config that we need for synthetic nerf perturbation?
Thanks!
Very excited this has been released.
Will it work on a M3 Max?
If not, any plans to support Macs?
I have run:
python -m train \
--gin_configs=configs/zipnerf/360.gin \
--gin_bindings="Config.data_dir = '${DATA_DIR}'" \
--gin_bindings="Config.checkpoint_dir = '${CHECKPOINT_DIR}'"
python -m eval \
--gin_configs=configs/zipnerf/360.gin \
--gin_bindings="Config.data_dir = '${DATA_DIR}'" \
--gin_bindings="Config.checkpoint_dir = '${CHECKPOINT_DIR}'"
python -m render \
--gin_configs=configs/zipnerf/360.gin \
--gin_bindings="Config.data_dir = '${DATA_DIR}'" \
--gin_bindings="Config.checkpoint_dir = '${CHECKPOINT_DIR}'" \
--gin_bindings="Config.render_dir = '${CHECKPOINT_DIR}/render/'" \
--gin_bindings="Config.render_path = True" \
--gin_bindings="Config.render_path_frames = 240" \
--gin_bindings="Config.render_video_fps = 30"
python scripts/zipnerf/generate_tables_360.py
Now I have 4 video's in the render folder that was created.
The output says there are 2 videos missing as certain files were not created during the process.
I0122 20:01:14.943336 140617853252672 render.py:110] Rendered in 11.274s
I0122 20:01:15.526105 140617853252672 render.py:138] Creating videos.
I0122 20:01:15.769567 140617853252672 videos_utils.py:95] Video shape is (840, 1297)
I0122 20:01:15.769668 140617853252672 videos_utils.py:134] Making video /home/user/home/user/Documents/workspace/camp_zipnerf/checkpoints/360_v2_garden_1297x840//render/path_renders_step_200000/videos/360_v2_garden_1297x840_checkpoints_path_renders_step_200000_color.mp4...
I0122 20:01:29.177273 140617853252672 videos_utils.py:132] Images missing for tag normals
I0122 20:01:29.177362 140617853252672 videos_utils.py:132] Images missing for tag normals_rectified
I0122 20:01:29.177408 140617853252672 videos_utils.py:134] Making video /home/user/home/user/Documents/workspace/camp_zipnerf/checkpoints/360_v2_garden_1297x840//render/path_renders_step_200000/videos/360_v2_garden_1297x840_checkpoints_path_renders_step_200000_acc.mp4...
I0122 20:01:32.633051 140617853252672 videos_utils.py:134] Making video /home/user/home/user/Documents/workspace/camp_zipnerf/checkpoints/360_v2_garden_1297x840//render/path_renders_step_200000/videos/360_v2_garden_1297x840_checkpoints_path_renders_step_200000_distance_mean.mp4...
I0122 20:01:49.162443 140617853252672 videos_utils.py:134] Making video /home/user/home/user/Documents/workspace/camp_zipnerf/checkpoints/360_v2_garden_1297x840//render/path_renders_step_200000/videos/360_v2_garden_1297x840_checkpoints_path_renders_step_200000_distance_median.mp4...
All 4 videos are just blank back screens for a few seconds, files seem corrupt according to Github as you can see below. The images in the render folder show stuff, but also probably not as desired. I don't know what went wrong. The dataset and colmap output has been used thoroughly with other repo's without issues.
Thanks for the great work and code release. Currently trying out a test with the gardenvase dataset.
Not an "issue" with your code to report at the moment, but in this repo you do not refer to some of the datasets which are used in the examples, namely the NYC apartment, Alameda home, etc. Those are restricted?
Congrats on the impressive works! I believe that they will inspire numerous future works.
Also, sorry for the unrelated question, but I would like to know whether there are any plans to release the large-scale dataset showed in the Zip-NeRF teaser video and described in SMERF (berlin, nyc, alameda, london)?
Thanks!
Hi,
Thank you so much for your work.
I am testing and get nice rendering results.
However, it will be wonderful if I can render and check using an interactive viewer similar to nerfstudio. Is there any viewer which help me to do that, or guide me how to do?
Thank you in advance and hope get your help.
Hi, I hope you don't mind, but could you tell me if any of the errors/warnings underneath, selected and copied during the process, are problematic and/or potentially do hurt the performance of your code on my system?
E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I0124 13:07:20.764660 140638609667136 xla_bridge.py:660] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
I0124 13:07:20.768469 140638609667136 xla_bridge.py:660] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
Warning: image_path not found for reconstruction
I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
And a last question: the output of generate_tables_360.py looks like this in my terminal, formatting is off due to only one result in my set?
Average Results:
\hline
Our Model & \cellcolor{red}28.30 & \cellcolor{red}0.864 & 8.02
Per-Scene Results:
psnr
& \textit{360_v2_garden_1297x840/checkpoints} \\\hline
\hline
Our Model & \cellcolor{red}28.30
ssim
& \textit{360_v2_garden_1297x840/checkpoints} \\\hline
\hline
Our Model & \cellcolor{red}0.864
Thanks for the authors' outstanding work. However, due to JAX's high dependency on CUDA, the default environment is only compatible with CUDA 11.8 and CUDA 12.2. Here, I provide an environment that can be applied to CUDA 11.3-CUDA 11.7.
conda create -n camp_zipnerf python=3.10
conda activate camp_zipnerf
Then, pip install the following requirements:
numpy==1.26.3
jax==0.4.6
jaxlib==0.4.6
flax==0.6.1
opencv-python==4.9.0.80
pillow==10.2.0
tensorboard==2.10.1
tensorflow==2.10.0
gin-config==0.5.0
dm-pix==0.4.2
rawpy==0.19.0
mediapy==1.2.0
immutabledict==4.1.0
ml_collections
jaxcam==0.1.1
chex==0.1.7
Finally, it's necessary to install the CUDA version of jaxlib to enable GPU-accelerated training.
# python 3.10
wget https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.4.6+cuda11.cudnn82-cp310-cp310-manylinux2014_x86_64.whl
pip install jaxlib-0.4.6+cuda11.cudnn82-cp310-cp310-manylinux2014_x86_64.whl
# python 3.11
wget https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.4.6+cuda11.cudnn82-cp311-cp311-manylinux2014_x86_64.whl
pip intall jaxlib-0.4.6+cuda11.cudnn82-cp311-cp311-manylinux2014_x86_64.whl
After that, you may need to replace flax.core.copy
with flax.core.FrozenDict.copy
in internal/train_utils.py
to save checkpoints correctly.
Hi, thanks for this great paper and code.
After running camp/360_train.sh, I am trying to run 360_eval.sh or 360_render.sh, a flax.errors.ScopeParamShapeError always occur.
Is there a solution?
Below is the console log when running 360_render.sh.
`
2024-04-25 05:47:31.916448: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-04-25 05:47:31.916484: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-04-25 05:47:31.917323: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-04-25 05:47:32.577797: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I0425 05:47:41.449953 140104660227264 render.py:144] Rendering config:
eval/Config.adam_beta1 = 0.9
eval/Config.adam_beta2 = 0.99
eval/Config.adam_eps = 1e-15
eval/Config.arcore_format_pose_file = None
eval/Config.autoexpose_renders = False
eval/Config.batch_size = 8192
eval/Config.batching = 'all_images'
eval/Config.camera_perturb_dolly_use_average = False
eval/Config.camera_perturb_intrinsic_single = True
eval/Config.camera_perturb_sigma_dolly_z = 0.0
eval/Config.camera_perturb_sigma_focal_length = 0.0
eval/Config.camera_perturb_sigma_look_at = 0.0
eval/Config.camera_perturb_sigma_position = 0.0
eval/Config.camera_perturb_zero_distortion = False
eval/Config.cast_rays_in_eval_step = True
eval/Config.cast_rays_in_train_step = True
eval/Config.charb_padding = 0.001
eval/Config.checkpoint_dir =
'/home/user/camp_zipnerf_output/zipnerf/360/garden'
eval/Config.checkpoint_every = 10000
eval/Config.checkpoint_init = False
eval/Config.checkpoint_keep = 2
eval/Config.colmap_subdir = None
eval/Config.compute_disp_metrics = False
eval/Config.compute_normal_metrics = False
eval/Config.compute_procrustes_metric = False
eval/Config.data_coarse_loss_mult = 0.0
eval/Config.data_dir = '/home/user/data/360_v2/garden'
eval/Config.data_loss_mult = 1.0
eval/Config.data_loss_type = 'charb'
eval/Config.dataset_loader = 'llff'
eval/Config.debug_mode = False
eval/Config.deterministic_showcase = True
eval/Config.disable_multiscale_loss = False
eval/Config.disable_pmap_and_jit = False
eval/Config.distortion_loss_curve_fn =
(@math.power_ladder, {'p': -0.25, 'premult': 10000.0})
eval/Config.distortion_loss_mult = 0.01
eval/Config.distortion_loss_target = 'tdist'
eval/Config.donate_args_to_train = True
eval/Config.dtu_light_cond = 3
eval/Config.early_exit_steps = None
eval/Config.eikonal_coarse_loss_mult = 0.0
eval/Config.eikonal_loss_mult = 0.0
eval/Config.enable_grid_c2f = False
eval/Config.enable_loss_scaler = False
eval/Config.eval_crop_borders = 0
eval/Config.eval_dataset_limit = 2147483647
eval/Config.eval_only_once = True
eval/Config.eval_quantize_metrics = True
eval/Config.eval_raw_affine_cc = False
eval/Config.eval_render_interval = 1
eval/Config.eval_save_output = True
eval/Config.eval_save_ray_data = False
eval/Config.exposure_percentile = 97.0
eval/Config.factor = 4
eval/Config.far = 1000000.0
eval/Config.far_plane_meters = None
eval/Config.focal_length_var_loss_mult = 0.0
eval/Config.forward_facing = False
eval/Config.gc_every = 10000
eval/Config.grad_max_norm = 0.0
eval/Config.grad_max_val = 0.0
eval/Config.grid_c2f_weight_method = 'cosine_sequential'
eval/Config.image_subdir = None
eval/Config.jax_rng_seed = 20200823
eval/Config.llff_load_from_poses_bounds = False
eval/Config.llff_use_all_images_for_training = False
eval/Config.llffhold = 8
eval/Config.load_alphabetical = True
eval/Config.load_colmap_points = False
eval/Config.load_ngp_format_poses = False
eval/Config.lock_up = False
eval/Config.loss_scale = 1000.0
eval/Config.lr_delay_mult = 1e-08
eval/Config.lr_delay_steps = 20000
eval/Config.lr_final = 0.000125
eval/Config.lr_final_grid = None
eval/Config.lr_init = 0.00125
eval/Config.lr_init_grid = None
eval/Config.max_steps = 200000
eval/Config.multiscale_train_factors = None
eval/Config.near = 0.0
eval/Config.near_plane_meters = None
eval/Config.np_rng_seed = 20201473
eval/Config.num_border_pixels_to_mask = 0
eval/Config.num_showcase_images = 5
eval/Config.optimize_cameras = False
eval/Config.optimize_test_cameras = False
eval/Config.optimize_test_cameras_batch_size = 10000
eval/Config.optimize_test_cameras_for_n_steps = 200
eval/Config.optimize_test_cameras_lr = 0.001
eval/Config.orientation_coarse_loss_mult = 0.0
eval/Config.orientation_loss_mult = 0.0
eval/Config.orientation_loss_target = 'normals_pred'
eval/Config.param_regularizers =
{'grid_0': (0.1, @jnp.mean, 2, 1),
'grid_1': (0.1, @jnp.mean, 2, 1),
'grid_2': (0.1, @jnp.mean, 2, 1)}
eval/Config.patch_size = 1
eval/Config.predicted_normal_coarse_loss_mult = 0.0
eval/Config.predicted_normal_loss_mult = 0.0
eval/Config.principal_point_reg_loss_mult = 0.0
eval/Config.principal_point_var_loss_mult = 0.0
eval/Config.print_camera_every = 500
eval/Config.print_every = 100
eval/Config.rad_mult_max = 1.0
eval/Config.rad_mult_min = 1.0
eval/Config.radial_distortion_var_loss_mult = 0.0
eval/Config.randomized = True
eval/Config.rawnerf_mode = False
eval/Config.render_calibration_distance = 3.0
eval/Config.render_calibration_keyframes = None
eval/Config.render_camtype = None
eval/Config.render_chunk_size = 32768
eval/Config.render_delete_images_when_done = True
eval/Config.render_dir =
'/home/user/camp_zipnerf_output/zipnerf/360/garden/render/'
eval/Config.render_dist_adaptive = False
eval/Config.render_dist_percentile = 0.5
eval/Config.render_focal = None
eval/Config.render_looped_videos = False
eval/Config.render_path = True
eval/Config.render_path_file = None
eval/Config.render_path_frames = 480
eval/Config.render_resolution = None
eval/Config.render_rgb_only = False
eval/Config.render_rotate_xaxis = 0.0
eval/Config.render_rotate_yaxis = 0.0
eval/Config.render_spherical = False
eval/Config.render_spline_const_speed = False
eval/Config.render_spline_degree = 5
eval/Config.render_spline_fixed_up = False
eval/Config.render_spline_interpolate_exposure = False
eval/Config.render_spline_interpolate_exposure_smoothness = 20
eval/Config.render_spline_keyframes = None
eval/Config.render_spline_keyframes_choices = None
eval/Config.render_spline_lock_up = False
eval/Config.render_spline_lookahead_i = None
eval/Config.render_spline_meters_per_sec = None
eval/Config.render_spline_n_buffer = None
eval/Config.render_spline_n_interp = 30
eval/Config.render_spline_outlier_keyframe_multiplier = None
eval/Config.render_spline_outlier_keyframe_quantile = None
eval/Config.render_spline_rot_weight = 0.1
eval/Config.render_spline_smoothness = 0.03
eval/Config.render_video_crf = 18
eval/Config.render_video_exts = ('mp4',)
eval/Config.render_video_fps = 60
eval/Config.robust_loss_scale = 0.01
eval/Config.save_calibration_to_disk = False
eval/Config.scene_bbox = None
eval/Config.spline_interlevel_params = {'blurs': (0.03, 0.003), 'mults': 0.01}
eval/Config.train_render_every = 0
eval/Config.transform_poses_fn = None
eval/Config.use_exrs = False
eval/Config.use_identity_cameras = False
eval/Config.use_perturbed_cameras = False
eval/Config.use_tiffs = False
eval/Config.vis_decimate = 0
eval/Config.vis_num_rays = 16
eval/Config.visualize_every = 10000
eval/Config.vocab_tree_path = None
eval/Config.world_scale = 1.0
eval/Config.z_max = None
eval/Config.z_min = None
eval/Config.z_phase = 0.0
eval/Config.z_variation = 0.0
I0425 05:47:41.546334 140104660227264 xla_bridge.py:660] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
I0425 05:47:41.548914 140104660227264 xla_bridge.py:660] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
*** using 4x downsampled images
*** Finding COLMAP data
*** Constructing NeRF Scene Manager
Warning: image_path not found for reconstruction
*** Processing COLMAP data
*** Loaded camera parameters for 185 images
*** image names sorted alphabetically
*** Loading images from /home/user/data/360_v2/garden/images_4
*** Loaded 185 images from disk
*** Loaded EXIF data for 185 images
*** Constructed COLMAP-to-world transform.
*** Constructed 480 render poses via ellipse path
*** Constructed train/test split: #train=161 #test=24
*** LLFF successfully loaded!
*** split=DataSplit.TEST
*** #images/poses/exposures=24
*** #camtoworlds=480
*** resolution=(840, 1297)
I0425 05:49:08.581712 140104660227264 checkpoints.py:1062] Restoring orbax checkpoint from //home/user/camp_zipnerf_output/zipnerf/360/garden/checkpoint_200000
I0425 05:49:08.584176 140104660227264 checkpointer.py:164] Restoring item from /home/user/camp_zipnerf_output/zipnerf/360/garden/checkpoint_200000.
W0425 05:49:09.925947 140104660227264 transform_utils.py:229] The transformations API will eventually be replaced by an upgraded design. The current API will not be removed until this point, but it will no longer be actively worked on.
I0425 05:49:09.927694 140104660227264 transform_utils.py:286] The following keys are not loaded from the original tree after applying specified transforms: opt_state/0/count, opt_state/0/mu/params/MLP_0/Dense_0/bias, opt_state/0/mu/params/MLP_0/Dense_0/kernel, opt_state/0/mu/params/MLP_0/Dense_1/bias, opt_state/0/mu/params/MLP_0/Dense_1/kernel, opt_state/0/mu/params/MLP_1/Dense_0/bias, opt_state/0/mu/params/MLP_1/Dense_0/kernel, opt_state/0/mu/params/MLP_1/Dense_1/bias, opt_state/0/mu/params/MLP_1/Dense_1/kernel, opt_state/0/mu/params/MLP_2/Dense_0/bias, opt_state/0/mu/params/MLP_2/Dense_0/kernel, opt_state/0/mu/params/MLP_2/Dense_1/bias, opt_state/0/mu/params/MLP_2/Dense_1/kernel, opt_state/0/mu/params/MLP_2/Dense_2/bias, opt_state/0/mu/params/MLP_2/Dense_2/kernel, opt_state/0/mu/params/MLP_2/Dense_3/bias, opt_state/0/mu/params/MLP_2/Dense_3/kernel, opt_state/0/mu/params/MLP_2/Dense_4/bias, opt_state/0/mu/params/MLP_2/Dense_4/kernel, opt_state/0/mu/params/MLP_2/Dense_5/bias, opt_state/0/mu/params/MLP_2/Dense_5/kernel, opt_state/0/mu/params/MLP_2/Dense_6/bias, opt_state/0/mu/params/MLP_2/Dense_6/kernel, opt_state/0/mu/params/grid_0/grid_016, opt_state/0/mu/params/grid_0/grid_032, opt_state/0/mu/params/grid_0/grid_064, opt_state/0/mu/params/grid_0/grid_128, opt_state/0/mu/params/grid_0/hash_256, opt_state/0/mu/params/grid_0/hash_512, opt_state/0/mu/params/grid_1/grid_0016, opt_state/0/mu/params/grid_1/grid_0032, opt_state/0/mu/params/grid_1/grid_0064, opt_state/0/mu/params/grid_1/grid_0128, opt_state/0/mu/params/grid_1/hash_0256, opt_state/0/mu/params/grid_1/hash_0512, opt_state/0/mu/params/grid_1/hash_1024, opt_state/0/mu/params/grid_1/hash_2048, opt_state/0/mu/params/grid_2/grid_0016, opt_state/0/mu/params/grid_2/grid_0032, opt_state/0/mu/params/grid_2/grid_0064, opt_state/0/mu/params/grid_2/grid_0128, opt_state/0/mu/params/grid_2/hash_0256, opt_state/0/mu/params/grid_2/hash_0512, opt_state/0/mu/params/grid_2/hash_1024, opt_state/0/mu/params/grid_2/hash_2048, opt_state/0/mu/params/grid_2/hash_4096, opt_state/0/mu/params/grid_2/hash_8192, opt_state/0/nu/params/MLP_0/Dense_0/bias, opt_state/0/nu/params/MLP_0/Dense_0/kernel, opt_state/0/nu/params/MLP_0/Dense_1/bias, opt_state/0/nu/params/MLP_0/Dense_1/kernel, opt_state/0/nu/params/MLP_1/Dense_0/bias, opt_state/0/nu/params/MLP_1/Dense_0/kernel, opt_state/0/nu/params/MLP_1/Dense_1/bias, opt_state/0/nu/params/MLP_1/Dense_1/kernel, opt_state/0/nu/params/MLP_2/Dense_0/bias, opt_state/0/nu/params/MLP_2/Dense_0/kernel, opt_state/0/nu/params/MLP_2/Dense_1/bias, opt_state/0/nu/params/MLP_2/Dense_1/kernel, opt_state/0/nu/params/MLP_2/Dense_2/bias, opt_state/0/nu/params/MLP_2/Dense_2/kernel, opt_state/0/nu/params/MLP_2/Dense_3/bias, opt_state/0/nu/params/MLP_2/Dense_3/kernel, opt_state/0/nu/params/MLP_2/Dense_4/bias, opt_state/0/nu/params/MLP_2/Dense_4/kernel, opt_state/0/nu/params/MLP_2/Dense_5/bias, opt_state/0/nu/params/MLP_2/Dense_5/kernel, opt_state/0/nu/params/MLP_2/Dense_6/bias, opt_state/0/nu/params/MLP_2/Dense_6/kernel, opt_state/0/nu/params/grid_0/grid_016, opt_state/0/nu/params/grid_0/grid_032, opt_state/0/nu/params/grid_0/grid_064, opt_state/0/nu/params/grid_0/grid_128, opt_state/0/nu/params/grid_0/hash_256, opt_state/0/nu/params/grid_0/hash_512, opt_state/0/nu/params/grid_1/grid_0016, opt_state/0/nu/params/grid_1/grid_0032, opt_state/0/nu/params/grid_1/grid_0064, opt_state/0/nu/params/grid_1/grid_0128, opt_state/0/nu/params/grid_1/hash_0256, opt_state/0/nu/params/grid_1/hash_0512, opt_state/0/nu/params/grid_1/hash_1024, opt_state/0/nu/params/grid_1/hash_2048, opt_state/0/nu/params/grid_2/grid_0016, opt_state/0/nu/params/grid_2/grid_0032, opt_state/0/nu/params/grid_2/grid_0064, opt_state/0/nu/params/grid_2/grid_0128, opt_state/0/nu/params/grid_2/hash_0256, opt_state/0/nu/params/grid_2/hash_0512, opt_state/0/nu/params/grid_2/hash_1024, opt_state/0/nu/params/grid_2/hash_2048, opt_state/0/nu/params/grid_2/hash_4096, opt_state/0/nu/params/grid_2/hash_8192, opt_state/1/count
I0425 05:49:09.967541 140104660227264 checkpointer.py:166] Finished restoring checkpoint from /home/user/camp_zipnerf_output/zipnerf/360/garden/checkpoint_200000.
I0425 05:49:09.968436 140104660227264 render.py:61] Rendering checkpoint at step 200000.
/home/user/miniconda3/envs/camp_zipnerf/lib/python3.11/site-packages/jax/_src/xla_bridge.py:945: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code.
warnings.warn(
I0425 05:49:10.007920 140104660227264 render.py:96] Evaluating image 1/480
I0425 05:49:10.008137 140104660227264 models.py:1046] Rendering chunk 1/34
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/user/_NeRF_Test/camp_zipnerf/render.py", line 199, in
app.run(main)
File "/home/user/miniconda3/envs/camp_zipnerf/lib/python3.11/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/user/miniconda3/envs/camp_zipnerf/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
^^^^^^^^^^
File "/home/user/_NeRF_Test/camp_zipnerf/render.py", line 194, in main
render_config(config)
File "/home/user/_NeRF_Test/camp_zipnerf/render.py", line 155, in render_config
render_pipeline(config)
File "/home/user/_NeRF_Test/camp_zipnerf/render.py", line 99, in render_pipeline
rendering = models.render_image( # pytype: disable=wrong-arg-types # jnp-array
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/_NeRF_Test/camp_zipnerf/internal/models.py", line 1085, in render_image
chunk_renderings, _ = render_fn(rng, chunk_rays)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/_NeRF_Test/camp_zipnerf/internal/train_utils.py", line 770, in render_eval_fn
model.apply(
File "/home/user/_NeRF_Test/camp_zipnerf/internal/models.py", line 279, in call
ray_results = mlp(
^^^^
File "/home/user/_NeRF_Test/camp_zipnerf/internal/models.py", line 779, in call
raw_density, x = predict_density(means, covs, **predict_density_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/_NeRF_Test/camp_zipnerf/internal/models.py", line 733, in predict_density
x = density_dense_layer(self.net_width)(x)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/camp_zipnerf/lib/python3.11/site-packages/flax/linen/linear.py", line 235, in call
kernel = self.param(
^^^^^^^^^^^
flax.errors.ScopeParamShapeError: Initializer expected to generate shape (36, 64) but got shape (12, 64) instead for parameter "kernel" in "/MLP_0/Dense_0". (https://flax.readthedocs.io/en/latest/api_reference/flax.errors.html#flax.errors.ScopeParamShapeError)
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.
`
Hi, I have stumbled upon two problems when training CamP. I was using zipnerf/360.gin
and camp/camera_optim_perturbed.gin
.
The two Problems are:
module 'camp_zipnerf.internal.math' has no attribute 'normalize'
. I was able to fix this by changing math.normalize
to spin_math.normalize
here:
"Model" object has no attribute "grid_representation"
raised here: camp_zipnerf/camp_zipnerf/internal/train_utils.py
Lines 618 to 622 in d502249
Are these actually Bugs or am I missing something?
Hi there,
What are the configurations that need to be changed to get Zip-NeRF and CamP to work on the TPUs? I tried to use the default configs/zipnerf/360.gin
to train the Mip-NeRF 360 datasets with, but I get the following error and can't quite figure out how to disable the hash encoding. In that case, would it be better if I ran Mip-NeRF 360 on the TPUs instead? Thanks!
Traceback (most recent call last):
File "/home/tnguyen1/camp_zipnerf/train.py", line 557, in <module>
app.run(main)
File "/home/tnguyen1/.local/lib/python3.11/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/tnguyen1/.local/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
^^^^^^^^^^
File "/home/tnguyen1/camp_zipnerf/train.py", line 142, in main
model, state, render_eval_pfn, train_pstep, lr_fn = train_utils.setup_model(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tnguyen1/camp_zipnerf/internal/train_utils.py", line 828, in setup_model
model, variables = models.construct_model(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tnguyen1/camp_zipnerf/internal/models.py", line 444, in construct_model
init_variables = model.init(
^^^^^^^^^^^
File "/home/tnguyen1/camp_zipnerf/internal/models.py", line 279, in __call__
ray_results = mlp(
^^^^
File "/home/tnguyen1/camp_zipnerf/internal/models.py", line 779, in __call__
raw_density, x = predict_density(means, covs, **predict_density_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tnguyen1/camp_zipnerf/internal/models.py", line 680, in predict_density
raise ValueError('Hash Encodings should not be used on a TPU.')
ValueError: Hash Encodings should not be used on a TPU.
--------------------
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.```
Hello,
I'm encountering issues related to the dependencies between JAX and Orbax in my current setup. My environment uses CUDA 11.8, and after running pip install -e .
, I configured the GPU version of JAX with the command pip install -U "jax[cuda11]"
. The process of running ./scripts/run_all_unit_tests.sh
completed without errors. However, during the training and saving stages, I encountered the following error:
File "/path_to_anaconda/envs/my_env/lib/python3.11/site-packages/orbax/checkpoint/multihost/utils.py", line 90, in broadcast_one_to_some
in_tree = jax.tree.map(pre_jit, in_tree)
^^^^^^^^
File "/path_to_anaconda/envs/my_env/lib/python3.11/site-packages/jax/_src/deprecations.py", line 53, in getattr
raise AttributeError(f"module {module!r} has no attribute {name!r}")
AttributeError: module 'jax' has no attribute 'tree'
After investigating, I found that the code in orbax-checkpoint requires JAX version 0.4.6 or higher. However, when I manually upgraded JAX and JAXlib to version 0.4.6, I received the following error:
camp-zipnerf 0.0.2 requires jax==0.4.23, but you have jax 0.4.6 which is incompatible.
As a result, the training code could not run correctly:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/path_to_project/my_project/train.py", line 26, in <module>
import flax
File "/path_to_anaconda/envs/my_env/lib/python3.11/site-packages/flax/__init__.py", line 23, in <module>
from . import core
File "/path_to_anaconda/envs/my_env/lib/python3.11/site-packages/flax/core/__init__.py", line 15, in <module>
from .axes_scan import broadcast as broadcast
File "/path_to_anaconda/envs/my_env/lib/python3.11/site-packages/flax/core/axes_scan.py", line 22, in <module>
from jax.extend import linear_util as lu
ModuleNotFoundError: No module named 'jax.extend'
I would appreciate more detailed guidance on how to properly configure my environment. I have already tried using CUDA versions 12.3, 11.8, and even 11.6, but none of them worked successfully.
If possible, could you please provide some assistance or point me toward the correct setup steps?
Thank you in advance for your help!
Hi, I am trying to run the set up and I decided to run the unit tests. I see the following outputs where the tests have failed or skipped:
(camp_zipnerf) a@test-nerf-server:~/camp_zipnerf$ ./scripts/run_all_unit_tests.sh
Running camera_utils_test.py
./home/a/camp_zipnerf/internal/camera_utils.py:1178: RuntimeWarning: divide by zero encountered in divide
t1 = (corners[0] - ray_o) / ray_d
/home/a/camp_zipnerf/internal/camera_utils.py:1179: RuntimeWarning: divide by zero encountered in divide
t2 = (corners[1] - ray_o) / ray_d
..........
----------------------------------------------------------------------
Ran 11 tests in 19.966s
OK
Running coord_test.py
..s............F............s....s....s....s....s....s......s....s....s....s.....F....s....s.....
======================================================================
FAIL: test_construct_ray_warps_is_finite_and_in_range6 (0.5) (tests.coord_test.CoordTest)
tests.coord_test.CoordTest.test_construct_ray_warps_is_finite_and_in_range6 (0.5)
test_construct_ray_warps_is_finite_and_in_range(0.5)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/conda/envs/camp_zipnerf/lib/python3.11/site-packages/absl/testing/parameterized.py", line 323, in bound_param_test
return test_method(self, testcase_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/a/camp_zipnerf/tests/coord_test.py", line 93, in test_construct_ray_warps_is_finite_and_in_range
self.assertTrue(jnp.all(s_recon <= 1))
AssertionError: Array(False, dtype=bool) is not true
======================================================================
FAIL: test_pos_enc_20_0.2 (tests.coord_test.CoordTest)
tests.coord_test.CoordTest.test_pos_enc_20_0.2
test_pos_enc_20_0.2(20, 0.2)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/conda/envs/camp_zipnerf/lib/python3.11/site-packages/absl/testing/parameterized.py", line 321, in bound_param_test
return test_method(self, *testcase_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/a/camp_zipnerf/tests/coord_test.py", line 305, in test_pos_enc
self.assertLess(max_err, tol)
AssertionError: 0.23938593 not less than 0.2
----------------------------------------------------------------------
Ran 97 tests in 199.062s
FAILED (failures=2, skipped=13)
Running datasets_test.py
.
----------------------------------------------------------------------
Ran 1 test in 4.156s
OK
terminate called without an active exception
./scripts/run_all_unit_tests.sh: line 17: 70089 Aborted python -m unittest "$file"
Running geometry_test.py
........................
----------------------------------------------------------------------
Ran 24 tests in 4.341s
OK
Running geopoly_test.py
.........
----------------------------------------------------------------------
Ran 9 tests in 15.739s
OK
Running grid_utils_test.py
.........................................................................
----------------------------------------------------------------------
Ran 73 tests in 13.668s
OK
Running hash_resample_test.py
....
----------------------------------------------------------------------
Ran 4 tests in 2.613s
OK
Running image_utils_test.py
..........
----------------------------------------------------------------------
Ran 10 tests in 8.186s
OK
Running linspline_test.py
..s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s..
----------------------------------------------------------------------
Ran 145 tests in 15.251s
OK (skipped=29)
Running loss_utils_test.py
.......
----------------------------------------------------------------------
Ran 7 tests in 3.810s
OK
Running math_test.py
/home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
/home/a/camp_zipnerf/internal/math.py:340: RuntimeWarning: overflow encountered in square
return jnp.sign(x) * jnp.sqrt(1 - jnp.exp(-(4 / jnp.pi) * x**2))
/home/a/camp_zipnerf/internal/math.py:340: RuntimeWarning: overflow encountered in multiply
return jnp.sign(x) * jnp.sqrt(1 - jnp.exp(-(4 / jnp.pi) * x**2))
./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
....s....s....s....s....s....s....s....s....s....s....s......./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
.....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s.....s....s....s....s....s....s....s....s....s....s....s......./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
...s../home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: divide by zero encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: overflow encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: invalid value encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:299: RuntimeWarning: divide by zero encountered in divide
dn_true = scale / d
/home/a/camp_zipnerf/tests/math_test.py:299: RuntimeWarning: overflow encountered in divide
dn_true = scale / d
/home/a/camp_zipnerf/tests/math_test.py:303: RuntimeWarning: overflow encountered in multiply
dd_true = -scale * r / d
/home/a/camp_zipnerf/tests/math_test.py:303: RuntimeWarning: overflow encountered in divide
dd_true = -scale * r / d
./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: divide by zero encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: overflow encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: invalid value encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:299: RuntimeWarning: divide by zero encountered in divide
dn_true = scale / d
/home/a/camp_zipnerf/tests/math_test.py:299: RuntimeWarning: overflow encountered in divide
dn_true = scale / d
/home/a/camp_zipnerf/tests/math_test.py:303: RuntimeWarning: overflow encountered in multiply
dd_true = -scale * r / d
/home/a/camp_zipnerf/tests/math_test.py:303: RuntimeWarning: overflow encountered in divide
dd_true = -scale * r / d
.s/home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: divide by zero encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: overflow encountered in divide
r = np.float32(n) / np.float32(d)
/home/a/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: invalid value encountered in divide
r = np.float32(n) / np.float32(d)
...../home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
.../home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./home/a/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
....
----------------------------------------------------------------------
Ran 247 tests in 23.489s
OK (skipped=43)
Running quaternion_test.py
.......................................................................................................................................
----------------------------------------------------------------------
Ran 135 tests in 5.985s
OK
Running ref_utils_test.py
/home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
./home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s.../home/a/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
.s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s../home/a/camp_zipnerf/tests/ref_utils_test.py:68: RuntimeWarning: invalid value encountered in divide
xyz / np.sqrt(np.sum(xyz**2, axis=-1, keepdims=True))
..s/home/a/camp_zipnerf/tests/ref_utils_test.py:68: RuntimeWarning: invalid value encountered in divide
xyz / np.sqrt(np.sum(xyz**2, axis=-1, keepdims=True))
....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s..........
----------------------------------------------------------------------
Ran 648 tests in 25.003s
OK (skipped=128)
Running render_test.py
..............................................................
----------------------------------------------------------------------
Ran 62 tests in 47.985s
OK
Running resample_test.py
....../home/a/camp_zipnerf/internal/resample.py:128: UserWarning: Explicitly requested dtype float64 requested in zeros is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.
output = jnp.zeros((*locations.shape[:-1], data.shape[-1]), dtype=data.dtype)
/opt/conda/envs/camp_zipnerf/lib/python3.11/site-packages/jax/_src/numpy/array_methods.py:66: UserWarning: Explicitly requested dtype float64 requested in astype is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.
return lax_numpy.astype(arr, dtype)
........
----------------------------------------------------------------------
Ran 14 tests in 9.194s
OK
Running rigid_body_test.py
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 522 tests in 21.137s
OK
Running spin_math_test.py
........................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 328 tests in 21.759s
OK
Running stepfun_test.py
...................................................................
----------------------------------------------------------------------
Ran 67 tests in 60.031s
OK
Running train_utils_test.py
..
----------------------------------------------------------------------
Ran 2 tests in 1.513s
OK
Running utils_test.py
.......
----------------------------------------------------------------------
Ran 7 tests in 4.942s
OK
(camp_zipnerf) a@test-nerf-server:~/camp_zipnerf$
This is my machine:
(camp_zipnerf) a@test-nerf-server:~/camp_zipnerf$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
Can you tell the reason for this failure and will this effect the final result?
Hi!
Thank you for your great work!
Is there any way to get a 3D model of the result to watch it in Blender for example.
Thank you.
Hi, first of all thanks for this great paper and code. I finally managed to get your code to work!!
First, I tried to a task of replicating Table 1 in your paper for Single-Scale mipnerf360 dataset(7samples (bicycle,bonsai,counter,garden,kitchen,room,stump)), comparing with to without CamP.
Unfortunately, the average value of psnr:30.68,ssim:0.89(without CamP) degraded to psnr:27.43,ssim:0.82(with Camp)!
I used zipnerf/360_train.sh, zipnerf/360_eval.sh and zipnerf/360_render.sh, only rewriting DATA_DIR and CHECKPOINT_DIR, for w/o CamP.
And I used camp/360_train.sh, zipnerf/360_eval.sh and zipnerf/360_render.sh, rewriting DATA_DIR and CHECKPOINT_DIR and add option "-gin_configs=configs/camp/camera_optim.gin" for zipnerf/360_eval.sh and zipnerf/360_render.sh , for with CamP.
Unit test is all OK(with JAX), and Single-Scale mip-NeRF 360 Dataset Results of zipnerf(psnr,ssim) are roughly the same or better.
What do you think could be the reason it isn't working?
Is there any straightforward way to visualize the training result by moving the camera manually instead of rendering videos?
Thank you in advance.
I have run a test with the gardenvase dataset, did train, evaluate and then rendered the 480 frames, to then end abruptly, as it expected ffmpeg to be installed, which was not in the requirements. RuntimeError: Program 'ffmpeg' is not found
Rather not go through the +-90 minutes to render the frames again, I wondered if I can resume in some way from that point. Any advice? I now have a render-folder with 1920 files (4.3 GB) of generated data, for acc_, color_, distance_mean_ and distance_median_.
is it possible to create a webbased walkthrough in combination with camp_zipnerf and smerf-3d?
smerf-3d github project, webbased smerf viewer:
https://github.com/smerf-3d/smerf-3d.github.io
if yes, how to use the content and include it into the smerf viewer? are there any tutorials?
There are many warning coming while running the unit test any fix
/mnt/sdb1/sumant/zip_camp/camp_zipnerf$ ./scripts/run_all_unit_tests.sh
./mnt/sdb1/sumant/zip_camp/camp_zipnerf/camp_zipnerf/internal/camera_utils.py:1178: RuntimeWarning: divide by zero encountered in divide
t1 = (corners[0] - ray_o) / ray_d
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/camp_zipnerf/internal/camera_utils.py:1179: RuntimeWarning: divide by zero encountered in divide
t2 = (corners[1] - ray_o) / ray_d
............s.........................s....s....s....s....s....s......s....s....s....s..........s....s................................................................................................................................s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s........./mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/camp_zipnerf/internal/math.py:340: RuntimeWarning: overflow encountered in square
return jnp.sign(x) * jnp.sqrt(1 - jnp.exp(-(4 / jnp.pi) * x**2))
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/camp_zipnerf/internal/math.py:340: RuntimeWarning: overflow encountered in multiply
return jnp.sign(x) * jnp.sqrt(1 - jnp.exp(-(4 / jnp.pi) * x**2))
.....s....s....s....s....s....s....s....s....s....s....s......./mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
......s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s.....s....s....s....s....s....s....s....s....s....s....s..........s../mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: divide by zero encountered in divide
r = np.float32(n) / np.float32(d)
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: overflow encountered in divide
r = np.float32(n) / np.float32(d)
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: invalid value encountered in divide
r = np.float32(n) / np.float32(d)
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:299: RuntimeWarning: divide by zero encountered in divide
dn_true = scale / d
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:299: RuntimeWarning: overflow encountered in divide
dn_true = scale / d
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:303: RuntimeWarning: overflow encountered in multiply
dd_true = -scale * r / d
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:303: RuntimeWarning: overflow encountered in divide
dd_true = -scale * r / d
./mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: divide by zero encountered in divide
r = np.float32(n) / np.float32(d)
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: overflow encountered in divide
r = np.float32(n) / np.float32(d)
/mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:282: RuntimeWarning: invalid value encountered in divide
r = np.float32(n) / np.float32(d)
.s......../mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
./mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/math_test.py:46: RuntimeWarning: overflow encountered in cast
x = np.float32(np.exp(log_x))
.........................................................................................................................................../mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/ref_utils_test.py:89: RuntimeWarning: invalid value encountered in divide
grad_true = (xyz[:, 1] ** 2 + xyz[:, 2] ** 2) / (scale * denom**3)
..s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s../mnt/sdb1/sumant/zip_camp/camp_zipnerf/tests/ref_utils_test.py:68: RuntimeWarning: invalid value encountered in divide
xyz / np.sqrt(np.sum(xyz**2, axis=-1, keepdims=True))
..s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s....s............................................................................../mnt/sdb1/sumant/zip_camp/camp_zipnerf/camp_zipnerf/internal/resample.py:128: UserWarning: Explicitly requested dtype float64 requested in zeros is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.
output = jnp.zeros((*locations.shape[:-1], data.shape[-1]), dtype=data.dtype)
/mnt/sdb1/sumant/zip_camp/lib/python3.11/site-packages/jax/_src/numpy/array_methods.py:66: UserWarning: Explicitly requested dtype float64 requested in astype is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.
return lax_numpy.astype(arr, dtype)
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................://github.com/google/jax#current-gotchas..........
----------------------------------------------------------------------
Ran 2413 tests in 185.657s
OK (skipped=213)
Hey NeRF Community,
I am currently running the Zip-NeRF model and have successfully trained it to generate a video (.mp4) using my custom dataset. However, I would like to create a navigation experience similar to what is shown here: SMERF 3D Berlin Scene.
Does anyone know how I can achieve this?
Thanks!
Hey,
I have been looking forward to this code release for the past few months and now it is finally here! Thank you for your awesome work. The code looks really clean and I think it is great that you uploaded test set renderings for comparisons!
I have a few questions regarding the hash grid implementation.
triton_grids_inferface
that is used within grid_utils.py
? I might be missing something very simple here, sorry.precondition_scaling
?Looking forward to your answer!
Hi, first of all thanks for this grat paper and code.
I have a few questions regarding the paper:
camera_delta.py
it seems to me that the residuals are modelled as egocentric parameters. Is that correct or am I misunderstanding something here?I hope I am not completely misunderstanding something here. I would really appreciate if you could find the time to answer these question!
Hi, thanks for the great work. Just wondering if there are ways to let all cameras use shared intrinsic/distortion during the optimization? This can be quite useful for sparse view settings.
this requires as input in smerf to zipnerf models how do i feed this ? - https://github.com/google-research/google-research/blob/master/smerf/configs/zipnerf/london.gin
# london
smerf.internal.configs.Config.data_dir = "datasets/london"
smerf.internal.configs.Config.distill_teacher_ckpt_dir = "teachers/london"
# Downsample pixels
smerf.internal.configs.Config.factor = 2
# Render path config
**camp_zipnerf.internal.configs.Config.render_path_file = 'render_path_file.npy'**
camp_zipnerf.internal.configs.Config.render_resolution = (1392, 793) # (width, height)
camp_zipnerf.internal.configs.Config.render_focal = 606.465479 # in pixels
camp_zipnerf.internal.configs.Config.render_camtype = 'perspective'
Hi, thank you for making such a great achievement public!
This is not an issue, but a question. Can I run only CamP and obtain optimized camera poses?
I quickly checked the code, but it seems deeply integrated ZipNeRF and CamP. If there are any options which allow me to do it, it would be great if you could share it.
Thank you for your help in advance.
Hi, thanks for the wonderful job again.
I first conducted tests on the Mip-360 dataset, and the experimental results completely matched those in the paper and README. Subsequently, I tested the deep blending and tanks & Templates datasets used in 3D-GS.
The results for the DB (Deep Blending) dataset are as shown in the following table, and they represent the state-of-the-art rendering quality:
drjohnson | playroom | |
---|---|---|
PSNR | 30.150 | 30.946 |
SSIM | 0.90905 | 0.90808 |
LPIPS (VGG) | 0.21985 | 0.20522 |
However, when running the Tanks & Temples dataset, ZipNeRF appeared unable to converge, leading to highly fragmented outcomes. Theoretically, given that the format of the Tanks dataset is identical to that of DB, and considering the perfect results obtained on DB, this implies that my data loading process should be right. I am uncertain whether this issue pertains to the robustness of the ZipNeRF method itself or if the Tanks dataset necessitates a different configuration to achieve convergence.
I executed the following command:
DATA_DIR=/my/path/to/the/dataset
CHECKPOINT_DIR=./logs/zipnerf/tanks_db
SCENE=truck # train, drjohnson and playroom are in the same way
CUDA_VISIBLE_DEVICES=0 python -m train \
--gin_configs=configs/zipnerf/360.gin \
--gin_bindings="Config.data_dir = '${DATA_DIR}/${SCENE}'" \
--gin_bindings="Config.checkpoint_dir = '${CHECKPOINT_DIR}/${SCENE}'" \
--gin_bindings="Config.factor = 1"
Looking forward to your reply!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.