andyzeng / tsdf-fusion

Fuse multiple depth frames into a TSDF voxel volume.

Volumetric TSDF Fusion of Multiple Depth Maps

Update: a python version of this code with both CPU/GPU support can be found here.

CUDA/C++ code to fuse multiple registered depth maps into a projective truncated signed distance function (TSDF) voxel volume, which can then be used to create high quality 3D surface meshes and point clouds. Tested on Ubuntu 14.04 and 16.04.

Looking for an older version? See here.

Change Log

• Nov. 1, 2017. Bug fix: tsdf2mesh.m now properly generates a mesh in camera coordinates instead of voxel coordinates.
• Oct. 30, 2017. Notice: changed default weight threshold for SaveVoxelGrid2SurfacePointCloud in demo code to enable creating point cloud visualizations with only one depth frame.
• Aug. 30, 2017. Bug fix: remove deprecated offsets from surface distance compute during integration.

Requirements

• NVIDA GPU with CUDA support
• OpenCV (tested with OpenCV 2.4.11)

Demo

This demo fuses 50 registered depth maps from directory data/rgbd-frames into a projective TSDF voxel volume, and creates a 3D surface point cloud tsdf.ply, which can be visualized with a 3D viewer like Meshlab.

Note: Input depth maps should be saved in format: 16-bit PNG, depth in millimeters.

./compile.sh # compiles demo executable
./demo # 3D point cloud saved to tsdf.ply and voxel grid saved to tsdf.bin

[Optional] This demo also saves the computed voxel volume into a binary file tsdf.bin. Run the following script in Matlab to create a 3D surface mesh mesh.ply, which can be visualized with Meshlab.

tsdf2mesh; % 3D mesh saved to mesh.ply

Seen in

• 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (CVPR 2017)
• Semantic Scene Completion from a Single Depth Image (CVPR 2017)
• Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images (CVPR 2016)

References

• A Volumetric Method for Building Complex Models from Range Images (SIGGRAPH 1996)
• KinectFusion: Real-Time Dense Surface Mapping and Tracking (ISMAR 2011)
• Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images (CVPR 2013)

Citing

This repository is a part of 3DMatch Toolbox. If you find this code useful in your work, please consider citing:

@inproceedings{zeng20163dmatch, 
    title={3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions}, 
    author={Zeng, Andy and Song, Shuran and Nie{\ss}ner, Matthias and Fisher, Matthew and Xiao, Jianxiong and Funkhouser, Thomas}, 
    booktitle={CVPR}, 
    year={2017} 
}

tsdf-fusion's Issues

what cuda version

Hi,thank you for publishing your code.

Can i ask what version of CUDA are you using? I used 9.1 and it is encounter an fatal error with cudaMalloc when running demo.
My nvidia driver version is 384.111 and graphic processor is Quadro M2200.

Point cloud

Hello
I 'm just wondering is there a way to go back from Point cload to a TSDF volume ?

How can I integrate RGB data in TSDF?

Doesn't handle camera rotations proplery

I'm currently using the Realsense D415 camera to gather RGBD data of an object, specifically a bottle. This camera is attached to the end-effector of a 6DOF robot. I use the robot to gather real-world pose coordinates of the camera. I use this kinematics data as the pose of the camera for both translation and rotation. This program works if I perform translation across one axis. However, it doesn't work when I perform a rotation.

Reading the papers linked in the README, I can't seem to figure out if their pose data is relative to the world's origin or if they're absolute physical locations, such as ones received from a robot's kinematics data. Do you know if I should perform any additional transformations on my pose data, similar to how the C++ programs calculate the inverse of the base frame and multiply that with each subsequent frame?

The first image is with the camera just performing a translation across one axis. The second image is the camera performing a translation across the same axis, but also performing a 0 to 360 degree rotation along the way.

EDIT: By the way, I have double-checked the camera intrinsics, and have used two different realsense depth cameras and I'm encountering similar issues in both. Also, I have verified the positions of the end-effector (which is carrying the camera) are correct.

Error while compiling

Has someone faced with this issue while compiling?

99 errors detected in the compilation of "tmp/tmpxft 00001527 00000000-9 demo.spp1.ii".

Thank you.

questions on depth

hi, @andyzeng ，
what means on registered depth maps?? can i use the depth in the code as following

Pose-Data extraction from images

Hi, Im using intel realsense D415 CAMERA. I can extract the depth and rgb image ffrom the frame but I couldn't find a way to extract the frame.pose.txt mentioned in your dataset. Can you share, if you have a code to extract the .pose.txt file from every frame along with the rgb and depth images while the camera is capturing. It would be very helpful!!

result incomplete

hello @andyzeng
I want to reconstruct the sculpture，the input image is pictures of each side. but the output is always one side .how can I get the all side reconstruction. my result is below:
resource image:

result of one side:(input image is one side)

result of all side(input image is all side,)

Incorrect result

Hi . I have configured the code on Windows 10 + VS2013+ CUDA9.2+OpenCV3.0. And I run the demo successfully. But the result point cloud file(tsdf.ply) looks not correct. did there any body know why this happen?

I did not change the default parameters....

Here is the result.

question results

Hi,

I executed your code and your example. But why the results is incorrect ?

Thank you in advance.

question about camera intrinsics

Dear @andyzeng

Thank you for sharing your work.
I want to generate my own 3D volume data by changing some of your source code. However, I am confused by some parameters in this work. It will be appreciated that if you can give me some help.

First is the camera intrinsics. I did some research and found it is a 3x3 matrix. Then I looked through your work and found that the old and new version of this work are using different camera intrinsic matrix. Is it because you used different camera between these two work? In my own project, I'm using a Kinect as the depth camera. Can you tell me how can I set the camera intrinsics in my work?
The second is about the camera pose. It seems a 4x4 matrix and changes with every single depth camera. Can you tell me what is this parameter and how to set it?
The last is the input number. In your demo, you set the input number to 50. I changed the number to regenerate a voxel volume. However, when I changed the number to 1, the voxel volume turned to nothing. I am wondering can I generate a voxel volume from a single depth image? And is it the right way to change the input number to 1 if this work can generate a voxel volume from a single depth image?

Thant you

Sincerely yours
Tony

Support for correcting radial and tangential distortion

Is there something in the works for radial and tangential distortion correction? Since the Integrate function loops over the voxel grid, am I right in saying that implementing something like this would involve computing a mesh-grid of the image dimensions (640 x 480 for instance) and use that to compute the radial distance from the camera optical axis? You then only consider the point [pt_cam_x, pt_cam_y, pt_cam_z] if it is within the camera dimensions?

tsdf-fusion/demo.cu

Line 22 in ef59a08

for (int pt_grid_x = 0; pt_grid_x < voxel_grid_dim_x; ++pt_grid_x) {

Thanks

Format of pose.txt

@andyzeng
Sorry but could you please specify the format of camera poses in the data you use in demo code?
Because there're just 6 numbers in 6DoF format of pose or 7 numbers as in TUM RGB-D Trajectories Dataset but you have up to 16 params here (I noticed that 4 last ones is the same for all).
And I saw that most repos use 6DoF as camera poses, so if you have time then you can mark this as Pull request.
Thanks in advance!

question about projection of 3d point generated from TSDF

Hi @andyzeng
I'm trying to use your code (tsdf2mesh.m function) to generate the 3d point cloud.
From MSR 7-scene dataset, when I project the 3d point cloud generated by your function into image domain, it shows some misalignment. It could be my mistake (I considered extrinsic pose between depth and color camera).

Have you tried your function for MSR 7-scene dataset? or any suggestion for this kind of procedure?

depth_im should be initialized on heap

VC++ 15 on release mode is giving stack overflow if we initialize depth_im on stack

Segmentation fault while ./compile.sh

I had an error while compiling using ./compile.sh.

The error message:
Segmentation fault (core dumped)

Is there someone who had the same problem, and solved?

Problem when change to mesh

I use the tsdf2mesh.m to create a 3D surface mesh. I use the data in the project.
But when I run the tsdf2mesh.m,I wait 4 hours,my MATLAB is still busy.
What's the problem?
Thanks for your reply.

Depthmaps format

HI,
I have a depthmaps that looks like this: click

I've tried using them in your code in the form as show on the picture, and after making them grayscale, but generated point-cloud seems to be empty with those.

Any advices - how I should prepare pictures to make them work with your code ?

Regards.

How to use RGB image to add color information？

Hi，I want to build a map with RGB information，How to use RGB image to add color information？
Look forward to your reply,Thank you！

Troubles with compiling

Can this code work with Cuda 9.1 and Visual C++ 14.0 compiler?

Question about tsdf

Hi,

Is what I can calculate tsdf for a cloud of points not for depth images ?
You can help me with an idea

Thank you for your help.

absolute scale for reconstruction

@andyzeng hi,
how can I put the absolute scale to the result??

Problem for ".png" to ".bin"

@andyzeng Whether this is the method to get .bin file from .png file used in SSCNET or not?

how to add tsdf-fusion to rtabmap

I want to use rtabmap to rebuilt 3d models , so i want to add tsdf to improve the final result.

Problem with tsdf2mesh.m

Hi, I have generated the binary file and am trying to convert it to the ply file. But when I run the tsdf2mesh.m, an error shown below pops up:

Index in position 1 exceeds array bounds.

Error in tsdf2mesh (line 29)
meshPoints(1,:) = voxelGridOrigin(1) + points(2,:)*voxelSize; % x y axes
are swapped from isosurface

I check that the variable tsdf is not empty but the
fv = isosurface(tsdf, 0) would give empty results for fv, thus empty points and faces.

I am wondering how to solve this problem. Thanks!

redkitchen incomplete

Hi,

I tried reconstructing the whole redkitchen sequence, but the reconstruction didn't look like the one given in the dataset, some parts of the reconstruction are cropped/incomplete. Do you have any idea why?

The picture below is my reconstruction.

and this picture is the original one

code problem

In demo.cu function ::Integrate,the distant is got by the code float diff = (depth_val - pt_cam_z) * sqrtf(1 + powf((pt_cam_x / pt_cam_z), 2) + powf((pt_cam_y / pt_cam_z), 2)).

But i don't know why multiply sqrtf(1 + powf((pt_cam_x / pt_cam_z), 2) + powf((pt_cam_y / pt_cam_z), 2)).

Is there anyone could give me some help?Thanks a lot!

How to make reconstruction without matlab?

I find Pyhon version of tsdf not only have color but also can run without matlab. I have make my C++ version TSDF add colored , but I have no idea have to replace matlab. I know the key part is marching_cubes_lewiner (Python) and isosurface(matlab). Any wary to replace them in C++？ Any ideal and suggesion is OK !
Thanks!