orochippw / lightglue-onnxrunner Goto Github PK

Configuration
model: v0.1.3_end2end

python code: v0.1.3,
command is
python infer.py --img_paths assets/DSC_0410.JPG assets/DSC_0411.JPG --img_size 1024 --lightglue_path weights_test/superpoint_2048_lightglue_end2end.onnx --extractor_type superpoint --viz

result image is

c++ code: using master branch,
onnxruntime using onnxruntime-linux-x64-gpu-1.15.0

result image is

This is the result on the example image, the difference is more pronounced on my own real data. I would like to ask for the reasons for this deviation.
Is there any way to improve the accuracy to be as close as possible to the original pyhton code results? Thanks!

src/LightGlueOnnxRunner.cpp line 84,
cv::Mat resultImage = NormalizeImage(ResizeImage(tempImage ,cfg.image_size , scale , fn , interp)); , encounter error,
error: cannot bind non-const lvalue reference of type 'cv::Mat&' to an rvalue of type 'cv::Mat'.
Maybe it could be,

cv::Mat reszieImage = ResizeImage(tempImage ,cfg.image_size , scale , fn , interp);
cv::Mat resultImage = NormalizeImage(reszieImage);

Hi, thanks for sharing the code. Are you planning to have TensorRT support in C++? Thanks again.

[in LightGlueOnnxRunner.cpp]

First, thanks for the repo!

Spotted a delete that I think shouldnt be there.
You pass cfg by value, then delete a pointer to cfg.lightgluepath. This causes a double free error as the pass-by-value cfg object is implicitly deleted after function execution. The faulty line is line 63. Just remove delete modelPath.

PS: maybe the same thing is in LightGlueDecoupleOnnxRunner.cpp as well, idk

const size_t numOutputNodes = session->GetOutputCount();
        OutputNodeNames.reserve(numOutputNodes);
        for (size_t i = 0 ; i < numOutputNodes ; i++)
        {
            OutputNodeNames.emplace_back(_strdup(session->GetOutputNameAllocated(i , allocator).get()));
            OutputNodeShapes.emplace_back(session->GetOutputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape());
        }

        delete modelPath;
        
        std::cout << "[INFO] ONNXRuntime environment created successfully." << std::endl;

How do i convert to original coordinates? (keypoints_result)

Hope to add support for the Linux platform

Thanks for your works sharing!
I am a newcomers of deep learning deployment and i come across this problem shown below:

Hi, I'm a newbie and have this question: lightGlue's onnx model can support image inputs of different sizes, but how come the converted onnxRunner model only supports inputs like 512, 1024? ("Sample image size for ONNX tracing , resize the longer side of the images to this value. Supported image size {512 , 1024 , 2048}" in your code) What should I do if I want to achieve support for different sizes of inputs. I would appreciate it if you give a quick reply!

Hi! Thank you for making this project available!

What changes would need to be made to run on the gpu?

I am looking for around 20ms processing speed.

Thanks!

Hi, thank you for this cuda update and the decouple! Question, how can i return the cv::point AND the descriptor from the feature extraction?

Ideally a std vector of cv Keypoints and a cv mat of matching descriptors.

Thanks!

A strange error occurred while running your main.cpp. The matches have a strange shape: (395, -8646884894171381957). I am running the superpoint.onnx and superpoint_lightglue.onnx models

Could you share superpoint__lightglue_end2end.onnx? Thanks a lot!

[ERROR] ONNXRuntime environment created failed : Could not find an implementation for MultiHeadAttention(1) node with name 'MultiHeadAttention_0'

https://github.com/fabio-sim/LightGlue-ONNX/releases/tag/v1.0.0
Recently LightGlue-ONNX released there fused model which improve the performance substantially(~2x faster in my test) and change the format of the output tensor, making the current code not functional
output_tensors[2] is now the shape of (matchnum, 2) and the pairs are already computed(so you don't need to compute them yourself)
output_tensors[3] is now the shape of (matchnum), which denotes the matching score of each match

The following simple hack should do the work:

std::vector<int64_t> kpts0_Shape = output_tensors[0].GetTensorTypeAndShapeInfo().GetShape();
int64_t* kpts0 = (int64_t*)output_tensors[0].GetTensorMutableData<void>();
// 在Python里面是一个（batch = 1 , kpts_num , 2）的array，那么在C++里输出的长度就应该是kpts_num * 2
printf("[RESULT INFO] kpts0 Shape : (%lld , %lld , %lld)\n", kpts0_Shape[0], kpts0_Shape[1], kpts0_Shape[2]);

std::vector<int64_t> kpts1_Shape = output_tensors[1].GetTensorTypeAndShapeInfo().GetShape();
int64_t* kpts1 = (int64_t*)output_tensors[1].GetTensorMutableData<void>();
printf("[RESULT INFO] kpts1 Shape : (%lld , %lld , %lld)\n", kpts1_Shape[0], kpts1_Shape[1], kpts1_Shape[2]);

// matches: (match, [0, 1])
std::vector<int64_t> matches_Shape = output_tensors[2].GetTensorTypeAndShapeInfo().GetShape();
int64_t* matches0 = (int64_t*)output_tensors[2].GetTensorMutableData<void>();
int match_Counts = matches_Shape[0];
printf("[RESULT INFO] matches0 Shape : (%lld , %lld)\n", matches_Shape[0], matches_Shape[1]);

// match scores: (score)
std::vector<int64_t> mscore_Shape = output_tensors[3].GetTensorTypeAndShapeInfo().GetShape();
float* mscores = (float*)output_tensors[3].GetTensorMutableData<void>();

// Process kpts0 and kpts1
std::vector<cv::Point2f> kpts0_f, kpts1_f;
for (int i = 0; i < kpts0_Shape[1] * 2; i += 2)
{
    kpts0_f.emplace_back(cv::Point2f(
        (kpts0[i] + 0.5) / scales[0] - 0.5, (kpts0[i + 1] + 0.5) / scales[0] - 0.5));
}
for (int i = 0; i < kpts1_Shape[1] * 2; i += 2)
{
    kpts1_f.emplace_back(cv::Point2f(
        (kpts1[i] + 0.5) / scales[1] - 0.5, (kpts1[i + 1] + 0.5) / scales[1] - 0.5)
    );
}

std::set<std::pair<int, int> > matches;
std::vector<cv::Point2f> m_kpts0, m_kpts1;
for (int i = 0; i < matches_Shape[0] * 2; i += 2) {
    if (mscores[i/2] > cfg.threshold)
        matches.insert(std::make_pair(matches0[i], matches0[i+1]));
}

std::cout << "[RESULT INFO] matches Size : " << matches.size() << std::endl;

for (const auto& match : matches) {
    m_kpts0.emplace_back(kpts0_f[match.first]);
    m_kpts1.emplace_back(kpts1_f[match.second]);
}

keypoints_result.first = m_kpts0;
keypoints_result.second = m_kpts1;

std::cout << "[INFO] Postprocessing operation completed successfully" << std::endl;

Maybe you can add a flag for the new fused model, thanks!

Hi, OroChippw, Thanks for this amazing project! I have a problem about matcher time consume:
When I use dcouple SP + LG (superpoint.onnx + superpoint_lightglue.onnx) in my test, I save time consume to a txt file.
env: 3080 Laptop(8G), torch2.1, ubuntu20.04
I test 1000 pictures(1920x1080) on this env, matcher(LG) mean time consume is about 12ms. This result is very different from the result on your homepage, is my result correct? I can't believe it.
Hope your reply. thank u!