Homework4 of NTHU CVFX. Feature matching for moving forward and infinite zooming effect.
In below, we will use 4 types of input images to compare the results. The 4 types are depicted in below 4 columns:
Moving forward | Pitch rotation | Roll rotation | Yaw rotation | |
---|---|---|---|---|
img1 | ![]() |
![]() |
![]() |
![]() |
img2 | ![]() |
![]() |
![]() |
![]() |
Among the 4 types, the source image are all the same but the target images for feature matching are taken such that they micmic the camera moving forward, pitch rotation, roll rotation, yaw rotation respectively.
The source codes generating all the below results is feature_matcher.py
. We draw 50 matched features with the lowest distance for easier comparing the results by human eyes.
We compare between 4 feature descriptors:
- SIFT
- SURF
- SURF extended (128 dimensions instead of 64)
- ORB
and 2 matching algorithm settings:
- Brute Force Matching w/ Cross Check
- only accept those matches with value (i,j) such that i-th descriptor in set A has j-th descriptor in set B as the best match and vice-versa.
- Brute Force Matching w/ Ratio Test
- for each descriptor, find two closest neighbors.
- a descriptor is a match for its closest neighbor if the ratio of the distance of the closest neighbor to the distance of the second-closest neighbor is lower than a threshold.
- we set the ratio threshold to 0.7
Brute Force Matching w/ Cross Check | Brute Force Matching w/ Ratio Test | |
---|---|---|
SIFT | ![]() |
![]() |
SURF | ![]() |
![]() |
SURF extended | ![]() |
![]() |
ORB | ![]() |
![]() |
Brute Force Matching w/ Cross Check | Brute Force Matching w/ Ratio Test | |
---|---|---|
SIFT | ![]() |
![]() |
SURF | ![]() |
![]() |
SURF extended | ![]() |
![]() |
ORB | ![]() |
![]() |
Brute Force Matching w/ Cross Check | Brute Force Matching w/ Ratio Test | |
---|---|---|
SIFT | ![]() |
![]() |
SURF | ![]() |
![]() |
SURF extended | ![]() |
![]() |
ORB | ![]() |
![]() |
Brute Force Matching w/ Cross Check | Brute Force Matching w/ Ratio Test | |
---|---|---|
SIFT | ![]() |
![]() |
SURF | ![]() |
![]() |
SURF extended | ![]() |
![]() |
ORB | ![]() |
![]() |
- In general, ratio test works better than cross check in our experiment.
- ORB descripter works the best under all experiment settings.
- The results of SURF (64 dim) and SURF extended (128 dim) are similar.
- The effectiveness of the 4 descriptors under our experiment is: ORB > SIFT > SURF = SURF extended.
In many applications like panoram stitching and pose estimation, the matched keypoints is used to estimate the homography[ref] between the two images. In cases that the two images are captured only with different camera rotation and no camera translation, RANSAC can be applied to estimate the homography and remove false matched feature (see below figure).
However, if the two images are captured with different camera translation, only features on the same plane are guaranteed to match. In below example, most of the matched features are located on the farthest door. So after the homography estimation, the matched features closer to the camera are considered as mis-match. Some of them are really incorrect match (indicated as red) but it also remove some original correct match (indicated as green).
Before | After |
---|---|
![]() |
![]() |
To generate the moving forward effect from order images, we pick the first image as background and add image one by one as the camera move forward. The new image is scaled smaller and aligned with the previous one such that prominent feature are aligned. Below figure shows the result aligned by the estimated homography between two image:
![]() |
![]() |
![]() |
---|---|---|
![]() |
![]() |
![]() |
Finally, the exposure of each image is tuned respectively such that the result look more pleasing.