Render depth map about scaffold-gs

[ingra14m]

Hi @aiyb1314 , I have implemented a Differential Gaussian Rasterization with forward depth pass diff-gaussian-rasterization-extentions modified to Scaffold-GS. This is an independent repo that has the same contents with my another repo folked from official diff-gaussian-rasterization.

The only change you need to make is to include depth in the return values of gaussian_renderer/__init__.py:

rendered_image, radii, depth = rasterizer(
        means3D=xyz,
        means2D=screenspace_points,
        shs=None,
        colors_precomp=color,
        opacities=opacity,
        scales=scaling,
        rotations=rot,
        cov3D_precomp=None)

I did not include depth backward pass in that branch. It just outputs the depth. Feel free to use it.

[inspirelt]

Hi, two ways can generate the depth map:

1. Override the color with distance from xyz to camera_center and do rasterization:

xyz_dist = torch.norm(xyz - viewpoint_camera.camera_center, dim=-1, keepdims=True)
color = xyz_dist.repeat([1, 3])

2. Modify the CUDA rasterization according to https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files

[zsz-pro]

Hi, two ways can generate the depth map:

1. Override the color with distance from xyz to camera_center and do rasterization:

xyz_dist = torch.norm(xyz - viewpoint_camera.camera_center, dim=-1, keepdims=True)
color = xyz_dist.repeat([1, 3])

2. Modify the CUDA rasterization according to https://github.com/graphdeco-inria/diff-gaussian-rasterization/pull/5/files

I tried both methods mentioned above. The middle image corresponds to method 1, and the right image corresponds to method 2. These two are quite similar. However, the image on the left is obtained from a monocular depth model, and compared to that, neither of these methods produces correct depth maps. Here, I used cv2.COLORMAP_JET for the color space of depth mapping. Could you give me some guidance？ Thanks

[inspirelt]

'blue' denotes 'far' in the right two images, but denotes 'near' in the left image. Are there some misaligned rules?

[zsz-pro]

'blue' denotes 'far' in the right two images, but denotes 'near' in the left image. Are there some misaligned rules?

Thanks, I got it. The depth estimated by midas（the pretrained model I use) is the inverse depth maps