Slow first inference on jetson nano with dlr 1.2 about neo-ai-dlr

[trevor-m]

Hi @vizero1, the first inference is expected to take a long time because TensorRT will be building the engines.

In DLR 1.2 we added a feature to cache the engine to disk after it has been built so that next time you load the model it doesn’t need to be built again and the first inference will be fast. You can read about it here https://neo-ai-dlr.readthedocs.io/en/latest/tensorrt.html#caching-tensorrt-engines

Please see this page for more details on TRT options for DLR: https://neo-ai-dlr.readthedocs.io/en/latest/tensorrt.html

[vizero1]

@trevor-m thanks for the fast reply. I will try it out with the caching functionality but one thing is still not clear for me.
Why is there such a big difference between the different Jetpack versions? Is it due to the TRT versions?
Jetpack 4.3 with TRT 6 takes 5 minutes while Jetpack 4.4 with TRT 7 takes around 2 minutes?

[trevor-m]

Yes it is due to the TensorRT version. JetPack 4.4 uses TRT 7.1 which optimized engine build time. See “Builder layer timing cache” on this page https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-7.html#rel_7-1-3

[vizero1]

@trevor-m thanks.