magicleap / superpointpretrainednetwork

PyTorch pre-trained model for real-time interest point detection, description, and sparse tracking (https://arxiv.org/abs/1712.07629)

Research @ Magic Leap

SuperPoint Weights File and Demo Script

Introduction

This repo contains the pretrained SuperPoint network, as implemented by the originating authors. SuperPoint is a research project at Magic Leap. The SuperPoint network is a fully convolutional deep neural network trained to detect interest points and compute their accompanying descriptors. The detected points and descriptors can thus be used for various image-to-image matching tasks. For more details please see

• Full paper PDF: SuperPoint: Self-Supervised Interest Point Detection and Description

• Presentation PDF: Talk at CVPR Deep Learning for Visual SLAM Workshop 2018

• Authors: Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich

This demo showcases a simple sparse optical flow point tracker that uses SuperPoint to detect points and match them across video sequences. The repo contains two core files (1) a PyTorch weights file and (2) a python deployment script that defines the network, loads images and runs the pytorch weights file on them, creating a sparse optical flow visualization. Here are videos of the demo running on various publically available datsets:

Freiburg RGBD:

KITTI:

Microsoft 7 Scenes:

MonoVO:

Dependencies

• OpenCV python >= 3.4
• PyTorch >= 0.4

This repo depends on a few standard pythonic modules, plus OpenCV and PyTorch. These commands usually work (tested on Mac and Ubuntu) for installing the two libraries:

pip install opencv-python
pip install torch

Running the Demo

This demo will run the SuperPoint network on an image sequence and compute points and descriptors from the images, using a helper class called SuperPointFrontend. The tracks are formed by the PointTracker class which finds sequential pair-wise nearest neighbors using two-way matching of the points' descriptors. The demo script uses a helper class called VideoStreamer which can process inputs from three different input streams:

1. A directory of images, such as .png or .jpg
2. A video file, such as .mp4 or .avi
3. A USB Webcam

Run the demo on provided directory of images in CPU-mode:

./demo_superpoint.py assets/icl_snippet/

You should see the following output from the ICL-NUIM sequence snippet:

Run the demo on provided .mp4 file in GPU-mode:

./demo_superpoint.py assets/nyu_snippet.mp4 --cuda

You should see the following output from the NYU sequence snippet:

Run a live demo via webcam (id #1) in CPU-mode:

./demo_superpoint.py camera --camid=1

Run the demo on a remote GPU (no display) on 640x480 images and write the output to myoutput/

./demo_superpoint.py assets/icl_snippet/ --W=640 --H=480 --no_display --write --write_dir=myoutput/

Additional useful command line parameters

• Use --H to change the input image height (default: 120).
• Use --W to change the input image width (default: 160).
• Use --display_scale to scale the output visualization image height and width (default: 2).
• Use --cuda flag to enable the GPU.
• Use --img_glob to change the image file extension (default: *.png).
• Use --min_length to change the minimum track length (default: 2).
• Use --max_length to change the maximum track length (default: 5).
• Use --conf_thresh to change the point confidence threshold (default: 0.015).
• Use --nn_thresh to change the descriptor matching distance threshold (default: 0.7).
• Use --show_extra to show more computer vision outputs.
• Press the q key to quit.

BibTeX Citation

@inproceedings{detone18superpoint,
  author    = {Daniel DeTone and
               Tomasz Malisiewicz and
               Andrew Rabinovich},
  title     = {SuperPoint: Self-Supervised Interest Point Detection and Description},
  booktitle = {CVPR Deep Learning for Visual SLAM Workshop},
  year      = {2018},
  url       = {http://arxiv.org/abs/1712.07629}
}

Additional Notes

• We do not intend to release the SuperPoint training or evaluation code, please do not email us to ask for it.
• We do not intend to release the Synthetic Shapes dataset used to bootstrap the SuperPoint training, please do not email us to ask for it.
• We use bi-linear interpolation rather than the bi-cubic interpolation described in the paper to sample the descriptor as it is faster and gave us similar results.

Legal Disclaimer

Magic Leap is proud to provide its latest samples, toolkits, and research projects on Github to foster development and gather feedback from the spatial computing community. Use of the resources within this repo is subject to (a) the license(s) included herein, or (b) if no license is included, Magic Leap's Developer Agreement, which is available on our Developer Portal. If you need more, just ask on the forums! We're thrilled to be part of a well-meaning, friendly and welcoming community of millions.

superpointpretrainednetwork's Issues

the experiment between SuperPoint and ORB

@ddetone
Hi, I did the experiment between SuperPoint and ORB together with my classmate(three different groups data were used), strangely found that ORB's result is not so bad compared with SuperPoint(while in the paper, M. Score, a parameter indicates the overall performance, is much higher than ORB). In terms of
difference of matched point's y-axis, while the portion of the delta y (y2-y1) equals to zero is higher than SuperPoint for each-two test images.

Interest Point Detector Loss

Above is the interest point detector loss from the paper. I am unclear about what happens to the loss if we don't have a ground truth point within a X_hw cell. It would be create if you could clarify this for me.

Regards,
Patrick

Custom Point Tracking

How can I use the code to provide custom points for tracking in the whole video? Suppose I want only 4 points to be tracked in the whole video so how can I specify them in the code to customize it.

Improvising the speed

Apart from resizing the image and reducing the skin horn iteration threshold, what are the various ways to optimize the inference speed in older gpus?

demo wrong

When I just simply run the demo command as README.md :

./demo_superpoint.py assets/icl_snippet/

I got this:

Namespace(H=120, W=160, camid=0, conf_thresh=0.015, cuda=False, display_scale=2, img_glob='*.png', input='assets/icl_snippet/', max_length=5, min_length=2, nms_dist=4, nn_thresh=0.7, no_display=False, show_extra=False, skip=1, waitkey=1, weights_path='superpoint_v1.pth', write=False, write_dir='tracker_outputs/')
==> Processing Image Directory Input.
==> Loading pre-trained network.
==> Successfully loaded pre-trained network.
==> Running Demo.
Segmentation fault (core dumped)

As well as aboved problem, I use this test phase in my images, they finished unnormally like this:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
I debug and find the error location: when codes run at dmat = np.dot(desc1.T, desc2) in method nn_match_two_way, it will give such an error.
Anyone else see same situations?

How to create the .pth file?

How to create .pth file that has weights?

Code Source

It's amazing work. May you share the code? I would appreciate it !!!

time cost of superpoint feature extraction

Hi @ddetone , I'm curious about the time cost of superpoint feature extraction.

You mentioned in your paper that you estimate the total runtime of the system on a GPU
to be about 13ms. But when I extracted superpoint features, (640*480, with RGB 3 channel ), using single GPU (Tesla P100 PCIe 16GB), the time cost is about 60ms. Is that normal? Or have you calculated the time cost yourself and you may guess what's wrong with my 60ms.

Thanks in advance! Really hope to get your reply!

About pixel shuffle

Do you use the pixel shuffle in the superpoint model? if not, can you try to reinterpret the following sentence's meaning ?

In sentence in the superpoint paper:
Unfortunately, upsampling layers tend to add a high amount of computation and can introduce unwanted checkerboard artifacts, thus we designed the interest point detection head with an explicit decoder1 to reduce the computation of the model. This decoder has no parameters and is known as “sub-pixel convolution” or “depth to space” in TensorFlow or “pixel shuffle” in PyTorch.

thank you.

PointTracker:Warning,no points were added to tracker.

When I run the demo with nyu_snippet.mp4 in GPU mode,it takes a large amount of time to pre-train and displays warning: PointTracker: no points were added to tracker. And there's no Superpoint shown in visualisation.What 's the problem?

PyTorch 1.4+ and grid_sample behavior

Starting in version 1.4.0 of PyTorch the default behavior of torch.nn.functional.grid_sample was changed to align_corners=False. This might affect the outputting descriptors for certain input image sizes when compared to PyTorch 1.3-. To fix it, one simply needs to modify L281 as follows:

SuperPointPretrainedNetwork/demo_superpoint.py

Line 281 in 1fda796

desc = torch.nn.functional.grid_sample(coarse_desc, samp_pts)

desc = torch.nn.functional.grid_sample(coarse_desc, samp_pts, align_corners=True)

Binary descriptor

Is it somehow possible to convert/train the float descriptor to binary?
My pipeline is fully optimized for HAMMING distance matching so it would be nice to have the binary descriptor out of superpoint.

thanks,
Francesco

Missing two dashes from line 581 in the first argument of the add_argument() method

Recommend going from:
parser.add_argument('input', type=str, default='',

To:
parser.add_argument('--input', type=str, default='',

repeatability on HPatches Dataset

Hi,
I computed repeatability score of your pre-trained network using the repeatability code provided at https://github.com/rpautrat/SuperPoint/blob/master/superpoint/evaluations/detector_evaluation.py
I am getting the following results:

Whereas in paper, the reported repeatability score is 0.652 and 0.503 on illumination and viewpoint scenes respectively.

How to get feature descriptor for a patch?

Hi,
I wanted to get the HPatches benchmark score for SuperPoint. But to eliminate detector related factors, I want to have a feature vector describing 65x65 patches from the dataset.

How can I use the provided pre-trained model to get the descriptor on a 65x65 image patch?

Rubustness against Rotation

seem like that the pretrained model privided is not rubust agsinst Rotation,Right？

Results on rotated images

@ddetone my results after trying to retrain your model based on your released code:

It is able now to cope with rotation angles greater than 45 degree.

The threshold I was setting is 0.3
I modified your training a bit and trained the descriptor part only. ( 1000 coco images + 1000 hpatches images, 21 epochs)

Application of NMS

Hi I was trying to train on one of my dataset for a student research project. Can you please tell me if you applied NMS while homographic adaptation phase .

Thanks a lot

How to infer batch images at one time?

Hello, I have a question. How to infer batch images at one time ?

Anyone can give some advises?

Matching Score Metric - Hard to understand

Hello,

I'm trying to build my SuperPoint network from scratch and I want to implement the Matching Score metric. I am finding it hard to understand how the metric is computed. What do you mean by shared viewpoint region? Is the matching score the ratio of number of matches to number of keypoints detected in the two views?

Could you explain with an example from HPatches?

Thank you!

illumination change problem

Using superpoint demo to try to run one dataset of illumination changes in HPatchs, the number of feature points extracted is not consistent, have you considered solving this problem? And what do the blue lines that appear in the image represent?

train custum superpoint or superglue

How do i train my custum superpint and superglue models? Any git repo or package? Instead of using already available dataset?

Why the size of descriptor is 256 float ?

Thank you for your great work!
I wonder why the size of the descriptor is 256 float? Have you tested the other size ?

Thanks in advance! Really hope to get your reply!

how to create .pt file?

hello, I want to use this model with libtorch, can you teach me how to convert .pth file to .pt file?

About BatchNorm

sorry, I have a question about batchnorm layers. I didn't find norm layer in .py (demo_superpoint.py). like :
` # Shared Encoder.

x = self.relu(self.conv1a(x))

x = self.relu(self.conv1b(x))

x = self.pool(x)

x = self.relu(self.conv2a(x))

x = self.relu(self.conv2b(x))`

Onnx Inference

I'm using the following code to estimate the keypoints and matches using onnx

import json

import onnxruntime
import numpy as np
import cv2

path = "output/rgb.png"
img = cv2.imread(path)
img = cv2.resize(img, dsize=(640, 480), interpolation=cv2.INTER_AREA)
img.resize((1, 1, 640, 480))
data = json.dumps({'data': img.tolist()})
data = np.array(json.loads(data)['data']).astype('float32')
session = onnxruntime.InferenceSession("output/superpoint_640x480.onnx", None)
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name

print(input_name)
print(output_name)

result = session.run([output_name], {input_name: data})
print(result)

How do I interpret the result? or is it the proper way of doing it?

wrong keypoint coordinates

The keypoint coordinates are extracted from the heatmap via:

SuperPointPretrainedNetwork/demo_superpoint.py

Lines 251 to 257 in 1fda796

	xs, ys = np.where(heatmap >= self.conf_thresh) # Confidence threshold.
	if len(xs) == 0:
	return np.zeros((3, 0)), None, None
	pts = np.zeros((3, len(xs))) # Populate point data sized 3xN.
	pts[0, :] = ys
	pts[1, :] = xs
	pts[2, :] = heatmap[xs, ys]

The indices returned by np.where, respectively np.nonzero are in order of the dimensions. In a NumPy array, the first dimension is along the rows (i.e. the y coordinates in an image), the second dimension is along the columns (i.e. the x coordinates in an image).

Hence, syntactically this should be:

ys, xs = np.where(heatmap >= self.conf_thresh) # Confidence threshold.
# [...]
pts[0, :] = xs
pts[1, :] = ys
pts[2, :] = heatmap[ys, xs]

The actual variable name does not matter later, as those values are correctly normalised.

MagicPoint

Although the readme states that the synthetic shapes dataset or the training dataset, is there any plan to release the initial detector of MagicPoint?

Forgive me if this has been released already, or there's mention of it not being released elsewhere.

Any other training data except CoCo?

I found that your pretrained model performs much better than other open models on our images. May I ask if you have use other training data except coco? Thanks!