jonathonluiten / diff-gaussian-rasterization-w-depth Goto Github PK

Hi @JonathonLuiten , thanks for sharing your dynamic 3D gaussian work! May I ask the reason why you use median depth instead of the accumulated depth ( alpha blend the depth in the same way as color) to get the rendered depth image? I do find the former one achieves a cleaner depth map with less long tail effect, but I believe the latter is more friendly to back-propogate the gradient of depth to the gaussians, just curious about your motivation of making this decision choice for you dynamic gaussian splatting paper.

Hello,
Is it possible to render just single pixel which returns the color of each single ray from different colmap camera positions using depth/diff-gaussian-rasterization?

I am interested in getting final color at particular location for different view directions, by providing 3DGS pointcloud, location of target pixel for tracking in first fully rasterized image using first colmap camera pose from colmap pose sequence or 3D location in scene (how to find it?), and colmap camera poses.
Can you please guide me to achieve this?

Hi,

I want to ask what is exact definition of the depth. Is it the z-value depth? or the depth in the camera space after world_to_camera projection?

Many Thanks

Is this support drawing the Gaussian geometry in the image?

setup.py cannot be run because of a missing folder/library. Kindly clarify.

It seems that the viewmatrix (w2c) in GaussianRasterizationSettings is not differentiable. When traversing through grad_fn.next_functions.

            # l1 + dssim 
            loss.backward()

            grad_fn = loss.grad_fn
            while grad_fn is not None:
                print(grad_fn)
                grad_fn = grad_fn.next_functions[0][0]

output:

<AddBackward0 object at 0x7f97e9d962b0>
<AddBackward0 object at 0x7f97e9d96310>
<MulBackward0 object at 0x7f97e9d962b0>
<MeanBackward0 object at 0x7f97e9d96310>
<AbsBackward0 object at 0x7f97e9d962b0>
<SubBackward0 object at 0x7f97e9d96310>
<MaskedSelectBackward0 object at 0x7f97e9d962b0>
<torch.autograd.function._RasterizeGaussiansBackward object at 0x7f97e9d7c140>
<AccumulateGrad object at 0x7f97e9d962b0>

I couldn't find the gradient function for viewmatrix. Do I need to edit the backward function of GaussianRasterization to compute the gradient of the camera view? Are there any other methods available? Thanks.

Hi thanks you sharing this

I replaced diff_gaussian_rasterization of this repo with that in the vanilla gaussian

It perfectly renders RGB images but the depths are all white.

Any advice on how to handle this issue ?

Image RGB

Image depth

(splatam) root@carlos:/SplaTAM/diff-gaussian-rasterization-w-depth.git# python setup.py install
Traceback (most recent call last):
File "setup.py", line 13, in
from torch.utils.cpp_extension import CUDAExtension, BuildExtension
File "/usr/local/lib/python3.8/dist-packages/torch/init.py", line 191, in
_load_global_deps()
File "/usr/local/lib/python3.8/dist-packages/torch/init.py", line 153, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python3.8/ctypes/init.py", line 373, in init
self._handle = _dlopen(self._name, mode)
OSError: /opt/hpcx/ompi/lib/libmpi.so.40: undefined symbol: opal_hwloc201_hwloc_get_type_depth

Hi thanks for the work!
I am using splatam and want to install pypose which requires at least torch 1.13, however, the rasterization needs 1.12, hence there is a conflict.

If i want to run the splatam (import the gs package) with torch=1.13, do you know how i can manage it?
Thanks!