tau-j / rtmlib

RTMPose series (RTMPose, DWPose, RTMO, RTMW) without mmcv, mmpose, mmdet etc.

🙋 Tau

I am a researcher of Shanghai AI laboratory. Currently, I'm the maintainer of MMPose.

Research interests: building lightweight, real-time yet high-performance deeplearning solutions, and AI implementation and productization

• Human-centric perception tasks, focusing on human pose estimation (e.g. RTMPose, RTMO (CVPR2024 Accepted), RTMW)
• Large Language Models pretraining and SFT (e.g. InternLM/InternLM2)
• LLM As Agent (e.g. AgentLego)

Blog

 

🥳 🚀 My Projects

RTMPose

rtmlib's Issues

Issue Report: Low FPS with onnxruntime-gpu

Issue Report: Low FPS with onnxruntime-gpu

Description

I've installed the rtmlib library by following the "install from source code" instructions using pip install onnxruntime-gpu. I am using the cuda device and the onnxruntime backend. However, I am experiencing a maximum of 7 fps.

System Information

• onnxruntime-gpu Version: 1.16.3
• CUDA Availability: Yes
  • CUDA Version: 11.8
• cuDNN Availability: Yes
  • cuDNN Version: 8700
• nvcc (NVIDIA Cuda compiler) Version: 11.8.89
• GPU: NVIDIA GeForce RTX 3080 Ti Laptop GPU
• Operating System: Windows 11

Minimal Code to Reproduce

import time
import cv2
from rtmlib import PoseTracker, Wholebody, draw_skeleton

device = 'cuda'
backend = 'onnxruntime'

cap = cv2.VideoCapture('./dance.mp4')
openpose_skeleton = False  # True for openpose-style, False for mmpose-style

wholebody = PoseTracker(
    Wholebody,
    det_frequency=20,
    to_openpose=openpose_skeleton,
    mode='lightweight',  # balanced, performance, lightweight
    backend=backend,
    device=device)

frame_idx = 0
start_time = time.time()
while cap.isOpened():
    success, frame = cap.read()

    if not success:
        break
    s = time.time()
    keypoints, scores = wholebody(frame)
    det_time = time.time() - s
    print('det: ', det_time)

    img_show = frame.copy()

    # if you want to use black background instead of original image,
    # img_show = np.zeros(img_show.shape, dtype=np.uint8)

    img_show = draw_skeleton(img_show,
                             keypoints,
                             scores,
                             openpose_skeleton=openpose_skeleton,
                             kpt_thr=0.43)
    frame_idx += 1

    img_show = cv2.resize(img_show, (960, 540))
    cv2.imshow('img', img_show)
    cv2.waitKey(1)

end_time = time.time()
elapsed_time = end_time - start_time
fps = frame_idx / elapsed_time
print('Average FPS: {:.2f}'.format(fps))

Results

det:  0.048416852951049805
det:  0.04732680320739746
det:  0.043745994567871094
det:  0.05219006538391113
Average FPS: 6.73

Expected Behavior

I expect to achieve a higher FPS considering the use of onnxruntime-gpu and the provided system specifications. Any guidance on optimizing the performance would be appreciated. Thanks!

rtmo推理不正确

rtmo.py： 前处理部分应该去掉对图像的归一化；base.py：rtmo的openvino推理时候，好像不应该判断output0.ndim==ouput1.ndim

Can you provide a version that will run on windows?

关于模型加速的一些想法

请问，能否让目标检测模型det和姿态检测rtmpose共用一个backbone，将检测器的回归头和simcchead分别解耦，这样能否加快推理速度？

使用rtmdet推理结果不对

你好，我下载了mmdetection中的rtmdet-tiny模型，以及自己训练出来的rtmdet-tiny模型，转成onnx后，使用rtmlib推理出的结果总是不对。yolox的是对的。请问是哪里的问题呢？

How can I modify it so that it detects only the key points of a single person?