TensorRT cpp inference is slow. about tensorrt-for-yolo-series

[eshoyuan]

The image read uses about 10ms, so the inference still uses 30ms for INT8 engine.

[Linaom1214]

The image read uses about 10ms, so the inference still uses 30ms for INT8 engine.

which code ?

I think the infernce speed not related with engine which build by python script or trtexec tool.

[eshoyuan]

The image read uses about 10ms, so the inference still uses 30ms for INT8 engine.

which code ?

I think the infernce speed not related with engine which build python script or trtexec tool.

Sorry, I didn't express clearly. I didn't mean the difference between python script and trtexec tool. I mean I use trtexec to get the INT8 inference perfomance like this which is 3x faster than your code.

I use the code in cpp/end2end.

[Linaom1214]

The image read uses about 10ms, so the inference still uses 30ms for INT8 engine.

which code ?
I think the infernce speed not related with engine which build python script or trtexec tool.

Sorry, I didn't express clearly. I don't mean the difference between python script and trtexec tool. I mean I use trtexec to get the inference perfomance like this.

I use the code in cpp/end2end.

int main(int argc, char** argv) {
  if (argc == 5 && std::string(argv[1]) == "-model_path" && std::string(argv[3]) == "-image_path") {
    char* model_path = argv[2];
    char* image_path = argv[4];
    float* Boxes = new float[4000];
    int* BboxNum = new int[1];
    int* ClassIndexs = new int[1000];
    Yolo yolo(model_path);
    clock_t startTime, endTime;
    int num = 0;
    double total_time = 0;
    cv::Mat img;
    // warmup 
    while (num != 1000) {
      startTime = clock();
      img = cv::imread(image_path);
      yolo.Infer(img.cols, img.rows, img.channels(), img.data, Boxes, ClassIndexs, BboxNum);
      endTime = clock();
      double cur_timae = (double)(endTime - startTime) / CLOCKS_PER_SEC;
      total_time += cur_timae;
      num += 1;
    }
    // run inference
    auto start = std::chrono::system_clock::now();
    yolo.Infer(img.cols, img.rows, img.channels(), img.data, Boxes, ClassIndexs, BboxNum);
    auto end = std::chrono::system_clock::now();
    std::cout << std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count() << "ms" << std::endl;

    // cout << "The run time is:" << total_time / 1000 << "s" << endl;
    yolo.draw_objects(img, Boxes, ClassIndexs, BboxNum);

  } else {
    std::cerr << "--> arguments not right!" << std::endl;
    std::cerr << "--> yolo -model_path ./output.trt -image_path ./demo.jpg" << std::endl;
    return -1;
  }
}

you can use the code

I'm guessing it's just the warmup that affects the speed

[eshoyuan]

Oh, thank you! I forgot using warmup. I will test it tomorrow.

[Linaom1214]

Oh, thank you! I forgot using warmup. I will test it tomorrow.

This might look more simply, my test is correct

int main(int argc, char** argv) {
  if (argc == 5 && std::string(argv[1]) == "-model_path" && std::string(argv[3]) == "-image_path") {
    char* model_path = argv[2];
    char* image_path = argv[4];
    float* Boxes = new float[4000];
    int* BboxNum = new int[1];
    int* ClassIndexs = new int[1000];
    Yolo yolo(model_path);
    cv::Mat img;
    img = cv::imread(image_path);
    // warmup 
    for (int num =0; num < 10; num++) {
      yolo.Infer(img.cols, img.rows, img.channels(), img.data, Boxes, ClassIndexs, BboxNum);
    }
    // run inference
    auto start = std::chrono::system_clock::now();
    yolo.Infer(img.cols, img.rows, img.channels(), img.data, Boxes, ClassIndexs, BboxNum);
    auto end = std::chrono::system_clock::now();
    std::cout << std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count() << "ms" << std::endl;

    // cout << "The run time is:" << total_time / 1000 << "s" << endl;
    yolo.draw_objects(img, Boxes, ClassIndexs, BboxNum);

  } else {
    std::cerr << "--> arguments not right!" << std::endl;
    std::cerr << "--> yolo -model_path ./output.trt -image_path ./demo.jpg" << std::endl;
    return -1;
  }
}

[eshoyuan]

int main(int argc, char** argv) {
if (argc == 5 && std::string(argv[1]) == "-model_path" && std::string(argv[3]) == "-image_path") {
char* model_path = argv[2];
char* image_path = argv[4];
float* Boxes = new float[4000];
int* BboxNum = new int[1];
int* ClassIndexs = new int[1000];
Yolo yolo(model_path);
cv::Mat img;
img = cv::imread(image_path);
// warmup
for (int num =0; num < 10; num++) {
yolo.Infer(img.cols, img.rows, img.channels(), img.data, Boxes, ClassIndexs, BboxNum);
}
// run inference
auto start = std::chrono::system_clock::now();
yolo.Infer(img.cols, img.rows, img.channels(), img.data, Boxes, ClassIndexs, BboxNum);
auto end = std::chrono::system_clock::now();
std::cout << std::chrono::duration_caststd::chrono::milliseconds(end - start).count() << "ms" << std::endl;

// cout << "The run time is:" << total_time / 1000 << "s" << endl;
yolo.draw_objects(img, Boxes, ClassIndexs, BboxNum);

} else {
std::cerr << "--> arguments not right!" << std::endl;
std::cerr << "--> yolo -model_path ./output.trt -image_path ./demo.jpg" << std::endl;
return -1;
}
}

It works well! Thank you!