lzx551402 / geodesc

Implementation of ECCV'18 paper - GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints

GeoDesc implementation

TensorFlow implementation of GeoDesc for ECCV'18 paper "GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints", Zixin Luo, Tianwei Shen, Lei Zhou, Siyu Zhu, Runze Zhang, Yao Yao, Tian Fang and Long Quan.

Update 6/6/2021

Fix broken link to pretrained model.

Update 4/13/2020

The training code is released in a separate project, TFMatch, which also contains two follow-up works (ContextDesc, CVPR'19 and ASLFeat, CVPR'20).

Update 12/05/2019

Add three options (--pyr_off, --ori_off, --half_sigma) to help to better understand the performance. You may use the new test pair (test_img3.png, test_img4.png) to validate their influence.

--pyr_off (False by default): disable the pyramid construction so as to sample patches from the original image, which boosts the performance when scale change is not significant.

--ori_off (False by default): disable the orientation estimation by SIFT, which boosts the performance when the rotation change is not significant.

--half_sigma (True by default): halve the sigma value in pyramid construction, which empirically boosts the performance.

Some recent findings about GeoDesc: as described in the paper, we trained GeoDesc with random rotations up to 90°. Different from the setting in HardNet, we perturbed each training patch individually by a random angle uniformly sampled over [-90°, 90°], which leaded to a maximum of 180° rotation difference for an input patch pair. We have found such augmentation helped to acquire stronger rotational invariance, which is important in our target applications, i.e., 3D reconstruction from aerial images. However, in general cases where images are upright-aligned regarding their EXIF orientations, training with smaller rotation perturbation will give notably better performance.

Update 08/04/2019

We improve the patch cropping implementation and now it gets 5 times faster.

Update 04/12/2018

Part of the training data is released in GL3D. We will constantly update dataset to help to reproduce the work. Please also cite MIRorR if you find this dataset useful for your research.

Update 14/08/2018

We have provided an example to test the matching performance of GeoDesc (examples/image_matching.py). See usage below.

Requirements

Please use Python 2.7, install NumPy, OpenCV and TensorFlow. To run the image matching example, you may also need to compile opencv_contrib to get SIFT support.

Pre-trained model

Pre-trained GeoDesc model (in TensorFlow Protobuf format) can be found here.

Model in NumPy dictionary is available here, which is more handy to be converted to other formats.

Example scripts

1. Extract features of HPatches

An example script is provided to extract features of HPatches. HPatches should be ready in its original format.

After download HPatches, you can start to evaluate GeoDesc:

git clone https://github.com/lzx551402/geodesc.git
cd ../examples
python extract_features_of_hpatches.py \
    --hpatches_root=<hpatches_benchmark>/data/hpatches-release \
    --feat_out_path=<hpatches_benchmark>/data/descriptors

After the extraction, you can use HPatches benchmarking tools to evaluate GeoDesc (on split 'full' as GeoDesc is not trained on HPatches):

cd <hpatches_benchmark>/python
python hpatches_eval.py --descr-name='geodesc' \
    --task=verification --task=matching --task=retrieval --split=full

And then display the results:

python hpatches_results.py --descr-name='geodesc' \
    --task=verification --task=matching --task=retrieval --split=full --results-dir=

2. Test image matching

As described in the paper, the matching pipeline consists of: i) detect keypoints by SIFT detector, ii) crop patches in the scale space, iii) compute features on cropped patches, and iv) match the two images. If you want to achieve the efficiency reported in the paper and use it for large-scale matching tasks, we strongly suggest you implementing the pipeline in C++ with integrated GPU-based SIFT (e.g., SIFTGPU) and GPU-based matcher (e.g., OpenCV GPU matcher). We have provided here only a prototype for research purposes.

To get started, simply run:

cd examples
python image_matching.py --cf_sift

The matching results from SIFT (top) and GeoDesc (bottom) will be displayed. Type python image_matching.py --h to view more options and test on your own images.

(Image source: Graffiti sequence in Heinly benchmark)

Training code

The ground truth patches used to train GeoDesc are under preparation.

Benchmark on HPatches, mAP

Benchmark on Heinly benchmark

Benchmark on ETH local features benchmark

Application on 3D reconstructions

geodesc's Issues

Problems when calling the model trained by tfmatch

I used tfmatch training to generate a CKPT file. After it was converted into a Pb file, it cannot be used in the example in geodesc, and prompted:

• TypeError: Cannot interpret feed_dict key as Tensor: The name 'input:0' refers to a Tensor which does not exist. The operation, 'input', does not exist in the graph.

Is it because of the problem when I convert Pb files?or can you tell me the input nodes i can use.

running examples with tensorflow2

hello
I convert code to tensorflow2 but for loading savedmpdel.pb I need tag which is related MetaGraph . could you please help me?

ValueError: Importing a SavedModel with tf.saved_model.load requires a tags= argument if there is more than one MetaGraph. Got tags=None, but there are 0 MetaGraphs in the SavedModel with tag sets: []. Pass a tags= argument to load this SavedModel.

thank you

Extracting Descriptors on HPatches

Hi,

Your Geodesc implementation on different benchmarks is amazing.

But I have a question on extract_features_of_hpatches.py in terms of extracting descriptors from HPatches.

seq_data = load_seq(os.path.join(FLAGS.hpatches_root, hpatches_seq_list[i])) feat = sess.run("squeeze_1:0", feed_dict={"input:0": seq_data})

I suppose these two lines are about extracting Geodesc descriptors, right? But as HPatches mentioned, every patch is extracted using scale invariant interest point detecotrs, such as DoG. And then every patch is normalized to 65*65 pixels.

So you extract exactly 1 descriptor on every patch? How did you do that? What if there is more than 1 descriptor generated? Or maybe no descriptor can be generated?

Thanks in advance.

Best regards,
Weibo Qiu.

Match ratio

Hi，thank you for your sharing!It is a nice job!
Here is my little question:

 deep_good_matches, deep_mask = matcher_wrapper.get_matches(
                                   deep_feat1, deep_feat2, cv_kpts1, cv_kpts2, ratio=0.80, cross_check=True, info='deep')
sift_good_matches, sift_mask = matcher_wrapper.get_matches(
                                       sift_feat1, sift_feat2, cv_kpts1, cv_kpts2, ratio=0.89, cross_check=True, info='sift')

The ratio between the deep and sift method should de be same, why different?

why is this sorted according to cv_kpts.size?

if down_octave:
sort_idx = (-npy_kpts[:, 2]).argsort()
Hello, why is this sorted according to npy_kpts[:, 2]? I said that the content of npy_kpts[:, 2] is the diameter information of the feature point.Thanks！ @lzx551402

Reconstruction Pipeline

Hello, thanks for your sharing!

How could I apply GeoDesc to SIFT-based reconstruction pipeline like you did in your paper Fig.5. I'm new in program, any advice would be appreciated.

License information

Hello @lzx551402 ,

Thanks for sharing an implementation of your ECCV 2018 paper. Could you please provide license information for this project? Thanks

关于特征点使用的问题

patches的提取采样，我看代码里面要进行相似性变换，是不是如果只有keypoints但没有orientation等信息就没法用了

Patch size

Hi!
Sorry for disturbing you. I am a bit confused about the patch size setting in your paper.In L2-Net and HardNet,patch size is 32X32,but your patches seem to be 12σ × 12σ. I have seen LIFT. But I still do not fully understand the relationship between patch size in pixel and σ.Could you give me some suggestions？

Hpatches graph ploting script

Hi, thanks for you paper and code.
Can you tell me how to use the matlab script to generate the graph on hpatches?
When I use run_article.m and res_article.m in sequence, I get a wrong retrieval value.

thanks for your help

Training code

Hi,

Will you provide the training code to replicate the results from the paper and generate the frozen Tensorflow graph model?

Thanks

How to creating training dataset?

Hello!
I have a question about this, are you creating the training data set manually?
Or is it done in code?
Thanks