ghixu / acmp

Planar Prior Assisted PatchMatch Multi-View Stereo

ACMP

[News] The code for ACMH is released!!!
[News] The code for ACMM is released!!!
[News] The code for ACMMP is released!!!

About

This repository contains the code for the paper Planar Prior Assisted PatchMatch Multi-View Stereo, Qingshan Xu and Wenbing Tao, AAAI2020. If you find this project useful for your research, please cite:

@article{Xu2020ACMP,  
  title={Planar Prior Assisted PatchMatch Multi-View Stereo}, 
  author={Xu, Qingshan and Tao, Wenbing}, 
  journal={AAAI Conference on Artificial Intelligence (AAAI)},
  year={2020}
}
@article{Xu2019ACMM,  
  title={Multi-Scale Geometric Consistency Guided Multi-View Stereo}, 
  author={Xu, Qingshan and Tao, Wenbing}, 
  journal={Computer Vision and Pattern Recognition (CVPR)},
  year={2019}
}

Dependencies

The code has been tested on Ubuntu 14.04 with GTX Titan X.

• Cuda >= 6.0
• OpenCV >= 2.4
• cmake

Usage

• Complie ACMP

cmake .  
make

• Test

Use script colmap2mvsnet_acm.py to convert COLMAP SfM result to ACMP input   
Run ./ACMP $data_folder to get reconstruction results

Results on high-res ETH3D training dataset [2cm]

Mean	courtyard	delivery_area	electro	facade	kicker	meadow	office	pipes	playgroud	relief	relief_2	terrace	terrains
79.81	86.57	85.04	86.83	69.88	77.01	64.88	75.81	71.13	71.14	84.46	84.16	90.14	90.50

Acknowledgemets

This code largely benefits from the following repositories: Gipuma and COLMAP. Thanks to their authors for opening source of their excellent works.

acmp's Issues

unused code or bug?

Hi, I find a confusing code here, is this an unused code or a bug?

ACMP/ACMP.cu

Line 588 in 604e234

} while (depth_perturbed < params.depth_min && depth_perturbed > params.depth_max);

this situation seems never happens. But if I change the code to while (depth_perturbed < params.depth_min || depth_perturbed > params.depth_max); the program will get stuck.

Quesiton about Equation (9) in ACMP paper P(Xij | Zj, Theta_i)

https://arxiv.org/pdf/1912.11744.pdf

In your Graphical model of view selection, when you define the probability of having Xij given Zj and Theta_i, it seems to be independent from Zj. Since mij corresponds to the cost of a given Theta hypothesis

In this case the view selection probability (Eq 11) is reduced to being proportional to P(Zj | Znj)

I thus also don't get your Monte-Carlo sampling to define the weight for each source image Ij as wj.

What am I missing ?

Many thanks

Memory Requirements

Hey!

Are there any memory requirements for this codebase? I'm working with (~4000) 918x1368 images and often find the codebase exists with "Killed" indicating that I am running out of memory (128G) at some random point during the geometric consistency phase. Is this related to #4?

Depth map evaluation problem

I have read your paper and you conducted a depth map evaluation inside. I also use ETH3D training set data for depth map evaluation, but there are two issues that plague me:

1. The depth map provided by ETH only has sparse points with depth, but this is not the case in your paper. How do you process the true values of the depth map?
2. The depth map provided by ETH has distortion, but the training set data we used has no distortion, which leads to a significant difference between the results of my depth map evaluation using ACMMP and those in the paper.
   May I ask if you could tell me how to process the true values of depth maps? Thank you.

Bug Report: an illegal memory access was encountered.

In acmp.cu - CheckerboardPropagation()
line 808:
int pointTemp = left_near - (1 + i) - i * width;
should be:
int pointTemp = left_near - (1 + i) + i * width;

The code builds but produces blank results

Hi, thank you for open-sourcing this great work.

I have no problem in building the code, but when I try to run it the results are an image full of zero for both costs and depths. I tried to look at the code but it seems ok. The weird thing is that the Delaunay triangulation part selects all the pixels as support points (as you can see attached) since all the costs are zero and satisfy the condition . I tried also to remove all the iterations in order to look only at the initialization results, and the problem is indeed that depths and costs get never initialized beyond zero.

Maybe there is something I'm missing here, but can you help me out? The dataset I'm using is pipes from ETH3D.

Thank you in advance.

I have some question about the specific triangulation method used in this paper? Is it Delaunay Triangulation in 2D?

an illegal memory access was encountered in....ACMP.cu

@GhiXu First of all, thank you for opening up the novel algorithm-ACMP！When I executed the acmp program, an error occurred, which caused the data processing to fail，as shown in the figure below：

The log displayed by the program seems to indicate that the memory of the GPU is not enough，and my computer GPU is RTX1070.If you can give some suggestions or solutions, I will be very grateful！The resolution of the image is 6000*4000.

Speed up fusion and patch match

Hi
I would like to thank the author who did this awesome research. I can get very good results.
Is there a way to speed up these patchmatch and fusion? Any reference or advice would be highly appreciated. Thank you

how to get depth_map with the format of h5py?

hi, could you tell me how to get the depth_map with the format of h5py? Thank you very much.

Depth map visualization problem

Thanks for your great work! I get excellent result on my indoor dataset.
But when I try to visualize the depth map, I find it's ugly compared with colmap depth map. I just normalize the depth map through (depthmap - depthmap.min)/(depthmap.max - depthmap.min). Is there anything better I can do?
colmap depth map:

ACMP depth map:

Planar model:

ACMMP algorithm on ETH3D dataset

Hi, in the paper Planar Prior Assisted PatchMatch Multi-View Stereo you say maybe you will combine ACMP with multi-scale strategy in the future, is the algorithm ACMMP the implementation? It looks like so from the name.

If you do not know about ACMMP, do you implement ACMP + ACMM yourself? Will it be much better than ACMP? And how about the speed?

Unable to reproduce ETH3D results

Thank you for making your code available!

I am trying to use your ACMP code to reproduce your reconstruction results that are posted on the ETH3D website and that were published in your AAAI-20 paper. My methodology is:

1. Build the ACMP code from this repository (including PR #4)
2. Download the High-res multi-view training datasets (multi_view_training_dslr_undistorted.7z) from the ETH3D website
3. Use colmap2mvsnet_acm.py with default parameters to convert the COLMAP files to MVSNet format
4. Run the ACMP executable to reconstruct 3D point clouds
5. Run ETH3DMultiViewEvaluation to compute the F₁ score at 2 cm tolerance

When I use your code to reconstruct and evaluate the point clouds in this manner, I get some F₁ scores that differ significantly from those posted on the ETH3D website:

Dataset	F1 score (GitHub code)	F1 score (ETH3D website)	Δ
courtyard	84.09	85.61	-1.52
delivery	85.51	85.68	-0.17
electro	86.23	85.61	+0.62
facade	69.06	69.59	-0.53
kicker	72.83	77.33	-4.50
meadow	60.53	71.00	-10.47
office	74.17	74.45	-0.28
pipes	68.95	69.16	-0.21
playground	69.23	71.88	-2.65
relief	84.74	83.15	+1.59
relief_2	84.21	83.37	+0.84
terrace	90.19	89.92	+0.27
terrains	88.67	90.57	-1.90

I realize that the algorithm is non-deterministic, and that F₁ scores may vary slightly from one execution to the next. However, the F₁ scores for the kicker and meadow datasets are much worse when I run this ACMP code, compared to the results posted on the ETH3D website.

What are the differences between the code posted in this repository and the code that was used to generate the results in the paper and posted on the ETH3D website?

about the fusion code

Hi GhiXu, thank you for your great work!
I tested the code and it really produce beautiful result! And Since I'm more familiar with colmap, so I modified your code to read colmap project directly, so I do not have to use the script to convert project type. I opened the project in https://github.com/CanCanZeng/ACMP_colmap, but I can delete it if you do not like it.

I looked into the depth fusion code, it's very similar to the original fusion code in colmap, and I set the parameters to the same, but the fused point cloud is much different, I can't find the cause. Have you find this situation?

The images I used is from https://github.com/openMVG/ImageDataset_SceauxCastle, and the SfM is done by colmap.
the depth map is produced by ACMP on original resolution. the point cloud fused by ACMP and by colmap is much different


the later is result from ACMP.
I think the key parameters are reproj_error, relative_depth_diff, angle and num_consistent, I set them equal to colmap defaul value. I do not mean that the point cloud produced by colmap is better, I just want to know why they are so different.