bei91 / yolov5-onnx-tensorrt

How to run yolov5 model using TensorRT.

yolov5-onnx-tensorrt

This Repos contains how to run yolov5 model using TensorRT.
The Pytorch implementation is ultralytics/yolov5.
Convert pytorch to onnx and tensorrt yolov5 model to run on a Jetson AGX Xavier.
Support to infer an image .
Support to infer multi images simultaneously.

Requirements

Please use torch>=1.6.0 + onnx==1.8.0 + TensorRT 7.0.0.11 to run the code

Code structure

networks code is network
demo code runs tensorrt implementation on Jetson AGX Xavier

├── networks
├── utils
├── models
├── demo
│   ├── demo.py
|   ├── demo_batch.py
|   ├── Processor.py
|   ├── Processor_Batch.py
|   ├── Visualizer.py
|   ├── classes.py
|   ├── yolov5_onnx.py
│   └── onnx_tensorrt.py

• convert yolov5 pytorch model to onnx
• convert yolov5 onnx model to tensorrt
• pre-process image
• run inference against input using tensorrt engine
• post process output (forward pass)
• apply nms thresholding on candidate boxes
• visualize results

---

Compile pytorch model to onnx

python yolov5_onnx.py --help
usage: yolov5_onnx.py [-h] [-w WEIGHTS] [-is IMG-SIZE] [-bs BATCH-SIZE]

compile Pytorch model to ONNX

optional arguments:
  -h, --help            show this help message and exit
  -m WEIGHTS, --weights WEIGHTS
                        pytorch model file location
  -is IMG-SIZE, --img-size IMG-SIZE
                        image size
  -bs BATCH-SIZE, --batch-size BATCH-SIZE
                        batch size

Compile onnx model to tensorrt

python onnx_tensorrt.py --help
usage: onnx_tensorrt.py [-h] [-m MODEL] [-fp FLOATINGPOINT] [-o OUTPUT]

compile Onnx model to TensorRT

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        onnx file location
  -fp FLOATINGPOINT, --floatingpoint FLOATINGPOINT
                        floating point precision. 16 or 32
  -o OUTPUT, --output OUTPUT
                        name of trt output file

Run demo to infer an image

python demo.py -image=./path -model=./path/model.trt

Run demo_batch to infer multi images simultaneously

python demo_batch.py -image=./path -model=./path/model.trt

---

Thanks

• https://github.com/ultralytics/yolov5
• https://github.com/wang-xinyu/tensorrtx
• https://github.com/SeanAvery/yolov5-tensorrt

yolov5-onnx-tensorrt's Issues

There is no models folder in the project.

There is no models folder in the project.

Where can I get the models folder?

Layer Slice_4 failed validation

         Thank you for your kindness sharing. I have encountered the bug below, I don't know which is Slice_4 and how can i  debug it. Really appreciated for your advice

[TensorRT] WARNING: onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
ERROR In node 165 (parseGraph): INVALID_GRAPH: Assertion failed: ctx->tensors().count(inputName)
num layers: 132
lastlayers: <tensorrt.tensorrt.ILayer object at 0x7f971cde59d0>
[TensorRT] ERROR: Slice_4: slice is out of input range, input dimensions = [4,3,608,608], start = [0,0,0,0], size = [1,3,320,640], stride = [1,1,2,1].
[TensorRT] ERROR: Layer Slice_4 failed validation
[TensorRT] ERROR: Network validation failed.

NVRTC error

I have ran this command
python3 onnx_tensorrt.py --model yolov5s.onnx --floatingpoint 16 --output yolov5s.trt

and got this problem:
Beginning ONNX file parsing
[TensorRT] WARNING: onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
num layers: 265
nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
Traceback (most recent call last):
File "onnx_tensorrt.py", line 52, in
engine = builder.build_cuda_engine(network)
RuntimeError: NVRTC error:

The Post processing seems like wrong?

It seems like the post processing shows an error. any ideas?

trtbin coco_yolov5-16.trt
[TensorRT] WARNING: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
image arg bus.jpg
(1, 3, 608, 608)
execution time: 0.07985496520996094
(3326976,)
Traceback (most recent call last):
  File "demo_batch.py", line 87, in <module>
    main()
  File "demo_batch.py", line 79, in main
    output, image_raw_batch = processor.detect(img_batch)
  File "/home/administrator/yolov5-onnx-tensorrt/demo/Processor_Batch.py", line 92, in detect
    out = out.reshape(self.output_shapes[i])
ValueError: cannot reshape array of size 831744 into shape (1,3,80,80,85)

onnx to trt

can't convert to onnx file

python yolov5_onnx.py -w last.pt -is 640 -bs 4
Namespace(batch_size=4, img_size=[640, 640], weights='last.pt')
torch.Size([4, 3, 640, 640])
Traceback (most recent call last):
File "yolov5_onnx.py", line 33, in
model = attempt_load(opt.weights, map_location=torch.device('cpu')) # load FP32 model
File "/home/taucher/workspace/yolov5-onnx-tensorrt/networks/experimental.py", line 144, in attempt_load
model.append(torch.load(w, map_location=map_location)['model'].float().fuse().eval()) # load FP32 model
File "/home/taucher/.virtualenvs/yolov5/lib/python3.6/site-packages/torch/serialization.py", line 607, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "/home/taucher/.virtualenvs/yolov5/lib/python3.6/site-packages/torch/serialization.py", line 882, in _load
result = unpickler.load()
File "/home/taucher/.virtualenvs/yolov5/lib/python3.6/site-packages/torch/serialization.py", line 875, in find_class
return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'models'