huzi96 / gaussian-pcloud-render

Code repo for paper "Low Latency Point Cloud Rendering with Learned Splatting", CVPRW 2024.

3D Gaussian based Point Cloud Renderer

Yueyu Hu, Ran Gong, Qi Sun, Yao Wang.

Code repo for paper "Low Latency Point Cloud Rendering with Learned Splatting", CVPR Workshop (AIS: Vision, Graphics and AI for Streaming), 2024.

[PDF] [supp] [Workshop]

Related work:

Yueyu Hu, Ran Gong, Yao Wang. "Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering", arXiv:2406.05915, 2024.

This development of this repo is largely helped by and depending on the following open-source projects:

Pointersect: https://github.com/apple/ml-pointersect

3D Gaussian Splatting: https://github.com/graphdeco-inria/gaussian-splatting

Dependencies

PyTorch

The code is tested with PyTorch == 1.12.1 and CUDA 11.3, on NVIDIA RTX 4080 Super. Install PyTorch with,

conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch

MinkowskiEngine

Please follow https://github.com/NVIDIA/MinkowskiEngine to install MinkowskiEngine. The following command might simply work,

sudo apt install build-essential python3-dev libopenblas-dev

pip install -U MinkowskiEngine --install-option="--blas=openblas" -v --no-deps

Others

pip install imageio open3d==0.16.0 opencv-python torch_scatter xatlas scikit-image scipy pyexr pytorch_msssim lpips

Install Diff Gaussian Rasterization Package

cd diff-gaussian-rasterization
MAKEFLAGS="-j8" pip install .

Run example

Example 1: Quantized (200K)

python simple_benchmark.py pcrender --dataset_root ./example/THuman-256 --scale_factor 256 --fov 45 --voxelized --id_list 0519

Example 2: Non-quantized (800K)

python simple_benchmark.py pcrender --dataset_root ./example/THuman-800K --scale_factor 448 --fov 45 --id_list 0519

Test with more data samples with a mesh dataset

We provide as script sample_point_cloud_from_mesh.py that samples point clouds from meshes for testing. Please refer to the help message by python sample_point_cloud_from_mesh.py -h for usage.