Why are there two separate models for local/global descriptor? about dh3d

[xingruiyang]

Following the previous post: I have found a way to add the node ['xyz_feat_att'] to the global model, however, its weights are not in the checkpoint file:

The following variables are in the graph, but not found in the checkpoint: detection_block_reliable/detec_conv0/W, detection_block_reliable/detec_conv0/b, detection_block_reliable/detec_conv0/bn/gamma, detection_block_reliable/detec_conv0/bn/beta, detection_block_reliable/detec_conv0/bn/mean/EMA, detection_block_reliable/detec_conv0/bn/variance/EMA, detection_block_reliable/detec_conv1/W, detection_block_reliable/detec_conv1/b, detection_block_reliable/detec_conv1/bn/gamma, detection_block_reliable/detec_conv1/bn/beta, detection_block_reliable/detec_conv1/bn/mean/EMA, detection_block_reliable/detec_conv1/bn/variance/EMA, detection_block_reliable/detec_conv2/W, detection_block_reliable/detec_conv2/b, detection_block_reliable/detec_conv2/bn/gamma, detection_block_reliable/detec_conv2/bn/beta, detection_block_reliable/detec_conv2/bn/mean/EMA, detection_block_reliable/detec_conv2/bn/variance/EMA, detection_block_reliable/detec_conv_fc/W, detection_block_reliable/detec_conv_fc/b

Does that mean the local/global modes are actually two separate models?

[JuanDuGit]

Hi xingruiy,

we provide two models here to evaluate them on two tasks, one for the local descriptor and detector and the other for the global descriptor. They are separate because we use different data splitting strategies for these two tasks for fair comparisons with other methods, you can find more details in our supplementary material Sec.4.2.

The global model does not contain the weights of detector because the detector is not required to compute global features. If you want to get the local and global descriptors in a single forward pass, you can load the global model first and then load the local one. An easy way to implement this is to use the ChainInit from tensorpack
For example, ChainInit([SaverRestore(global_model_path), SaverRestore(local_model_path)])

[xingruiyang]

Many thanks! @JuanDuGit
That's a very useful tip. I have managed to load both model files by simply giving the SmartInit a list of model file paths:

session_init=get_model_loader(['model1','model2']),

And it worked like a charm!
I still have another question that is related to this:
I know the local features are formatted as 'xyz' [0:3], 'feat' (4:131) and 'att' [131], my question is can attention be simply be interpreted as the saliency of related points? I.e. If I am going to select K salient points, can I simply rank all points from the highest attention to the lowest and select the top K points? Or do I need to run the single_nms (which I assume is how you actually select K salient points) function that you used to select K points in your script?

[JuanDuGit]

Hi xingruiy,

NMS is used to produce an even distribution of selected keypoints, which is preferable when running the registration in the next step. Based on my own experience, I would recommend not skipping this operation. You can consider adujsting the selection criteria to get different number of keypoints, for example min_response_ratio, nms_radius.

Also, note that in our implementation, 1 - attention is consider as the saliency of points. Hope this helps~

[xingruiyang]

Thanks a lot! @JuanDuGit That makes a lot more sense. I will close the issue now.