city-super / scaffold-gs Goto Github PK

Did u use the adaptive strategy in the original GS paper(clone and split)?
Thanks

Hi! I'm trying to train on scenes with higher resolution, but getting worse results. What changes/optimizations do you suggest to achieve better reconstruction on 2K or 4K images?

Thanks!

hi, thanks for your great work, is it possible to modify anchor points without mlp in order to change scaffold-gs to explicit model? in unity / UE or other applications
@tongji-rkr
Looking forward to your reply, thanks!

When I build the scaffold_gs environment,it had this mistake.
Can you help me?

hello, it's an excellent work. However, when I compile the render tools in the SIBR_viewer, I encounter a problem that I can not find the torch. Do you know how I can solve this problem? Thank you so much.

There is no .gitmodule file, I don't know which version of diff_gaussian_rasterization you are using. There seems to be a visible_filter function that you added on top of the original code.

this function cannot be found

Hi, sorry to bother you, I'm very inspired by your work, however, I encountered some problems when trying to reproduce the paper, so when will you release your code?

hi, thanks for your great work, i got the errors when using the prebuild viewer on window,
cuda 11.4 RTX3080

LINE 720, FUNC sibr::GaussianView::onRenderIBR A CUDA error occurred during rendering:an illegal memory access was encountered. Please rerun in Debug to find the exact line!

looking forward to your reply, thanks!

This project turned out really great, but the ply files don't seem to be the same as the usual splat files. The existing web viewers and Unity projects can't properly display them. When viewing it on the web viewer, SuperSplat, all the scenes are just black.

I ran the network viewer once my training is started but it didn't automatically pick the training. Do I need to pass any additional command line arguments?
SIBR_viewers/install/bin/SIBR_gaussianViewer_app doesn't seem to pick the training. I have the training in the same machine but SIBR app is in different directory.
Also for rendering, why we can't just use the SIBR viewer from original Gaussian splatting authors?

Training progress: 100%|__________________________________________________________| 30000/30000 [1:58:29<00:00, 4.22it/s, Loss=0.0295401]
2024-03-14 03:49:43,489 - INFO:
[ITER 30000] Evaluating test: L1 0.2103012849887212 PSNR 11.615882555643717
2024-03-14 03:50:12,150 - INFO:
[ITER 30000] Evaluating train: L1 0.01723341103643179 PSNR 32.359920501708984

Hi there,

Thank you for the great work.

I was testing with a custom dataset and the result looks super strange. It's all black. Any insight about this?

Thank you so much

涛涛人帅，技术又强！

thanks for your great work, some errors about building SIBR

Scaffold-GS/SIBR_viewers/src/projects/gaussianviewer/renderer/GaussianView.cpp:550:53: error: ‘visible_filter’ is not a member of ‘CudaRasterizer::Rasterizer’ 550 | CudaRasterizer::Rasterizer::visible_filter( | ^~~~~~~~~~~~~~ Scaffold-GS/SIBR_viewers/src/projects/gaussianviewer/renderer/GaussianView.cpp:556:68: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations] 556 | ak_pos_all.contiguous().data<float>(),

looking forward to your reply, thanks! @inspirelt

Thanks for your great work!

Did you meet the problem that the testing accuracy is much lower than the training one?
My reproduced test PSNR is only 19.75 (25.34 in the paper) while the training PSNR is the same as the paper.

I repeated the experiment many times but the problem always exists.

Looking for your reply.

Great work.
Could you provide some results for the effectiveness of the appearance embedding. When I eable it.

Great Work.
The paper shows some results in the dataset proposed in VR-NeRF. However, the training scripts are not found in the repo.
Could you provide the loading code or converter code for evaluating such dataset?

Thanks for the great work.

I tried to run the code in single_train.sh with colmap data.

However, I got the following error

  }
};

extern "C"
__launch_bounds__(512, 4)
__global__ void reduction_prod_kernel(ReduceJitOp r){
  r.run();
}
nvrtc: error: invalid value for --gpu-architecture (-arch)

Training progress:   0%|                                                   | 0/30000 [00:00<?, ?it/s]

What could be the problem?

Has anyone tried combining this approach with a prune step like in https://github.com/VITA-Group/LightGaussian?

thank your Scaffold-GS , i like it very much !

python train.py  ^
-s E:\AI\A07\240310\win_cuda118\input\TaiDi_1280_35=5x7 ^
-m E:\AI\A07\240310\win_cuda118\input\TaiDi_1280_35=5x7\output ^
--voxel_size 0.001 --update_init_factor 16 ^
--appearance_dim 0 --ratio 1


Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
D:\conda\envs\cuda118\lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
D:\conda\envs\cuda118\lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Loading model from: D:\conda\envs\cuda118\lib\site-packages\lpips\weights\v0.1\vgg.pth
found tf board
2024-03-25 02:53:13,816 - INFO: args: Namespace(sh_degree=3, feat_dim=32, n_offsets=10, voxel_size=0.001, 
update_depth=3, update_init_factor=16, update_hierachy_factor=4, use_feat_bank=False, 
source_path='E:\\AI\\A07\\240310\\win_cuda118\\input\\TaiDi_1280_35=5x7', 
model_path='E:\\AI\\A07\\240310\\win_cuda118\\input\\TaiDi_1280_35=5x7\\output', images='images', 
resolution=-1, white_background=False, data_device='cuda', eval=False, lod=0, appearance_dim=0, lowpoly=False, 
ds=1, ratio=1, undistorted=False, add_opacity_dist=False, add_cov_dist=False, add_color_dist=False, 
iterations=30000, position_lr_init=0.0, position_lr_final=0.0, position_lr_delay_mult=0.01, 
position_lr_max_steps=30000, offset_lr_init=0.01, offset_lr_final=0.0001, offset_lr_delay_mult=0.01, 
offset_lr_max_steps=30000, feature_lr=0.0075, opacity_lr=0.02, scaling_lr=0.007, rotation_lr=0.002, 
mlp_opacity_lr_init=0.002, mlp_opacity_lr_final=2e-05, mlp_opacity_lr_delay_mult=0.01, 
mlp_opacity_lr_max_steps=30000, mlp_cov_lr_init=0.004, mlp_cov_lr_final=0.004, mlp_cov_lr_delay_mult=0.01, 
mlp_cov_lr_max_steps=30000, mlp_color_lr_init=0.008, mlp_color_lr_final=5e-05, mlp_color_lr_delay_mult=0.01, 
mlp_color_lr_max_steps=30000, mlp_featurebank_lr_init=0.01, mlp_featurebank_lr_final=1e-05, 
mlp_featurebank_lr_delay_mult=0.01, mlp_featurebank_lr_max_steps=30000, appearance_lr_init=0.05, 
appearance_lr_final=0.0005, appearance_lr_delay_mult=0.01, appearance_lr_max_steps=30000, percent_dense=0.01, 
lambda_dssim=0.2, start_stat=500, update_from=1500, update_interval=100, update_until=15000, 
min_opacity=0.005, success_threshold=0.8, densify_grad_threshold=0.0002, convert_SHs_python=False, 
compute_cov3D_python=False, debug=False, ip='127.0.0.1', port=6009, debug_from=-1, detect_anomaly=False, 
warmup=False, use_wandb=False, test_iterations=[7000, 30000], save_iterations=[7000, 30000, 30000], quiet=False, 
checkpoint_iterations=[], start_checkpoint=None, gpu='0')

$CUDA_VISIBLE_DEVICES
2024-03-25 02:53:13,825 - INFO: using GPU 0
2024-03-25 02:53:13,827 - INFO: save code failed~
2024-03-25 02:53:13,827 - INFO: Optimizing E:\AI\A07\240310\win_cuda118\input\TaiDi_1280_35=5x7\output
Output folder: E:\AI\A07\240310\win_cuda118\input\TaiDi_1280_35=5x7\output [25/03 02:53:13]
Reading camera 35/35 [25/03 02:53:13]
start fetching data from ply file [25/03 02:53:13]
Loading Training Cameras [25/03 02:53:13]
Loading Test Cameras [25/03 02:53:14]
Initial voxel_size: 0.001 [25/03 02:53:14]
Number of points at initialisation :  5579 [25/03 02:53:14]
Training progress:   0%|                                                                                                             | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last):
  
File "E:\AI\A07\240310\win_cuda118\train.py", line 536, in <module>
    training(lp.extract(args), op.extract(args), pp.extract(args), dataset,  args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, wandb, logger)
  
File "E:\AI\A07\240310\win_cuda118\train.py", line 136, in training
    voxel_visible_mask = prefilter_voxel(viewpoint_cam, gaussians, pipe,background)
  
File "E:\AI\A07\240310\win_cuda118\gaussian_renderer\__init__.py", line 203, in prefilter_voxel
    raster_settings = GaussianRasterizationSettings(

TypeError: <lambda>() missing 2 required positional arguments: 'kernel_size' and 'subpixel_offset'

Training progress:   0%|                                                                                                             | 0/30000 [00:00<?, ?it/s]

Hi,
thx for the source code of this wonderful project. I managed to train a model although I stopped after 7k iterations.
How to view the output? The regular SIBR viewer crashes. The one you are including does not build

CMake Error at src/CMakeLists.txt:79 (add_subdirectory):
add_subdirectory given source "core/view" which is not an existing
directory.

render.py gives: FileNotFoundError: [Errno 2] No such file or directory: 'outputs/truck/train/ours_7000/renders/00000.png'

Thx

self._local is not be used and need to save

Traceback (most recent call last):                                                                                                                                                                                                   
  File "/irip/liyihui_2022/job/Scaffold-GS/train.py", line 527, in <module>                                                                                                                                                          
    training(lp.extract(args), op.extract(args), pp.extract(args), dataset,  args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, wandb, logger)                          
  File "/irip/liyihui_2022/job/Scaffold-GS/train.py", line 187, in training                                                                                                                                                          
    torch.save((gaussians.capture(), iteration), scene.model_path + "/chkpnt" + str(iteration) + ".pth")                                                                                                                             
  File "/irip/liyihui_2022/job/Scaffold-GS/scene/gaussian_model.py", line 158, in capture                                                                                                                                            
    self._local,                                                                                                                                                                                                                      
AttributeError: 'GaussianModel' object has no attribute '_local'  

And I also found that self.demon is not be used and need to save

Traceback (most recent call last):                                                                                                                                                                                                   
  File "/irip/liyihui_2022/job/Scaffold-GS/train.py", line 527, in <module>                                                                                                                                                          
    training(lp.extract(args), op.extract(args), pp.extract(args), dataset,  args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, wandb, logger)                          
  File "/irip/liyihui_2022/job/Scaffold-GS/train.py", line 187, in training                                                                                                                                                          
    torch.save((gaussians.capture(), iteration), scene.model_path + "/chkpnt" + str(iteration) + ".pth")                                                                                                                             
  File "/irip/liyihui_2022/job/Scaffold-GS/scene/gaussian_model.py", line 158, in capture                                                                                                                                            
    self.denom,                                                                                                                                                                                                                      
AttributeError: 'GaussianModel' object has no attribute 'denom'                                             

Thank you for your outstanding work, does the code implement an interface for rendering depth maps?

The downloaded viewer folder does not contain a CMakeLists.txt file.

hello，it's a great work! and i want to konw how to get the Mem. when i run this work, i didn't get the Mem from the result.
Thanks

The training converges faster in terms of iterations, how about the actual speed (time)? Since you use mlps to decode the features (and the features are multi-res), I assume the training speed per iteration is slower than 3DGS. So how does the graph look like if we compare with 3DGS with time in x-axis and psnr in y-axis?

I am having a hard time understanding how the parameters for new anchors are given.

From what I have understood, the offsets are initialized as 0's.

But, I cannot understand how the new anchor's features (new_feat) is created.

• Also, why is new_scaling, new_rotation, new_opacities being created when mlp_cov and mlp_opacity is being used?

Thanks for your great work ！

When I run your code with torch==1.10.0+cu111, I encounter the error below. However, mip-splatting runs fine. So, is this a version-related issue?

Thanks for a great paper.

I was wondering, how did you evaluate the numbers like PSNR, SSIM, etc. in your paper?

My question is how many test cases did you pull out of the total number of datasets to evaluate the numbers.

I ask because it doesn't seem to be directly mentioned in the paper.

I am trying to render on a webbased tool which expect the following keys from the original GS .ply file.

color = np.array( [ 0.5 + SH_C0 * v["f_dc_0"], 0.5 + SH_C0 * v["f_dc_1"], 0.5 + SH_C0 * v["f_dc_2"], 1 / (1 + np.exp(-v["opacity"])), ] )
Can you please comment what are the alternative keys in the .ply file saved by your method as I couldn't find the following (property float f_dc_0
property float f_dc_1
property float f_dc_2 )keys in your method's .ply file?

Hi, I see that the appearance embedding is constructed based on the number of views in train cameras, and when shifting to eval mode, the uid of the test camera is directly used to query the learned embedding.

If I understand the appearance embedding correctly, it is set up so that the view-dependent effect can be better encoded; but since the test cameras and train cameras have different views, and so their uids have different meaning in this aspect, I think querying the same learned embedding would lead to wrong effect ? Thanks

Thank you for your great work! The src/core under SIBR you provided does not have the view folder, but there is the add_subdirectory(core/view) code under cmakelist. I would like to know whether this view folder is necessary?

How to initialization with a random point cloud in my own datasets?

Hello!
I have tried to use Scaffold-GS to render scenes with Free Camera Trajectories from F2-NeRF dataset (https://totoro97.github.io/projects/f2-nerf/).

The rendering of train/test camera views works fine, but when I want to create new views from the side (changing T vector), a lot of floating gaussians appear.

Here example below:

Render	SfM point cloud

It is worth noting that the point cloud from sfm is quite sparse, but how to avoid the appearance of floaters for such scenes?

The original paper (3DGS) is 27.21. In your paper, it is 28.69. What processing did you do?

Hi, thanks for the great work and congrats for CVPR acceptance. I saw in the code that ob_dist and ob_view are directly concatenated with anchor feature and input to the MLPs. Would using positional encoding (like in NeRF) further improve the view-dependent appearance rendering? as it captures high frequency variations, thanks.

Hi,
This work is perfect, but the result cannot be displayed in those existing tools like supersplat and unity.
Do you have any ideal to deal with this problem?

Hi, thank you for your great work!

With your anchor growing and pruning design, do you think if it exists such a case that all neural gaussians from one anchor will be moved to the voxel of another anchor. In this case you will not prune the first anchor because the opacity values of the GS from that anchor are still large enough. Just like the picture from your paper, there are some anchors floating around the object. Or is this acceptable? Maybe I have understand it wrong, please correct me.

Thanks!

Nice work！
想请教一个比较通用的问题，写CUDA的时候，一般是用下面这种方法进行调试吗？请指点下~

I tried to use the compiled SIBR_viewer provided by you, and the SIBR_viewer compiled by myself, and both ended up with a white screen and a flash back. This is what it shows:
[SIBR] -- INFOS --: OpenGL Version: 4.6.0 NVIDIA 537.34[major: 4, minor: 6]
[SIBR] -- INFOS --: Dataset type:
Number of input Images to read: 257
Number of Cameras set up: 257
LOADSFM: Try to open C:\Users\lcl124252\Desktop\code\Scaffold-GS\data\mydataset\scene1/sparse/0/points3D.bin
Num 3D pts 65407
[SIBR] -- INFOS --: SfM Mesh 'C:\Users\lcl124252\Desktop\code\Scaffold-GS\data\mydataset\scene1/sparse/0/points3d.bin successfully loaded. (65407) vertices detected. Init GL ...
[SIBR] -- INFOS --: Init GL mesh complete
[SIBR] -- INFOS --: Loading 251516 Anchor points
[SIBR] -- INFOS --: opacity_mlp : 0
[SIBR] -- INFOS --: cov_mlp : 0
[SIBR] -- INFOS --: color_mlp : 0
[SIBR] -- INFOS --: embedding_appearance : 0

Thanks for sharing this nice code.
I am trying to reproducing the result on the synthetic nerf.
I found that there is no predefined configuration for it, so I used the default settings from sing_train.sh file like,

scene='nerf_synthetic/lego'
exp_name='lego'
voxel_size=0.01
update_init_factor=16
gpu=3

./train.sh -d ${scene} -l ${exp_name} --gpu ${gpu} --voxel_size ${voxel_size} --update_init_factor ${update_init_factor}

After 30k iterations I got 32.05 dB for the lego scene, which is 3.64 dB lower than that in your paper. I think the default setting I used is not optimal training options. Could you share your testing options?

thank your Scaffold-GS , i like it very much !!!

Which parameter modified to obtain finer structure and higher PNSR ?