Comments (11)
As mentioned in #12 , I got some problem with SSD TRT plan distribution on TRTIS. And yes, there are several custom plugins in the SSD sample such as NMS, Normalize, PriorBox and so on. While serving with the engine plan serialized from the SSD sample, it seems that TRTIS did not deserialize these part correctly.
It would be great if the TRT plugin mechanism were fully supported in TRTIS. And I think a complete tutorial with SSD TRT plan might be helpful, as detection task is quite common in these days.
from server.
Has anyone tried using libnvinfer_plugin.so
in LD_PRELOAD
? I am unable to get it to work even though it has all the plugins.
Still stuck at
tensorrt_1 | E0601 15:02:09.585748 34 logging.cc:43] getPluginCreator could not find plugin Normalize_TRT version 1 namespace
tensorrt_1 | E0601 15:02:09.585764 34 logging.cc:43] Cannot deserialize plugin Normalize_TRT
and similar loglines for NMS_TRT and PriorBox_TRT
from server.
As a work-around for TRTIS not yet having TRT plugin support, you should be able to use LD_PRELOAD in many cases. For example, assuming your TRT plugins are compiled into trtplugins.so.
$ LD_PRELOAD=trtplugins.so trtserver --model-store=/tmp/models ...
You can load multiple plugin libraries with LD_PRELOAD. The limitation of this approach is that the plugins must be managed separately from the model store itself. And more seriously, if there are plugin name conflicts across multiple plugin libraries there is no way to handle it. But if you have just a single plugin library this should get you unblocked.
from server.
How would you suggest I compile plugins into trtplugins.so?
from server.
The TensorRT documentation has description and examples on how to create TensorRT plugin libraries: https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#extending
from server.
I am still facing the same issue:
LD_PRELOAD=libnvinfer_plugin.so /opt/tensorrtserver/bin/trtserver --model-store=/ScriptsNModels/TestModelZoo
E0612 20:28:44.454662 46 logging.cc:43] getPluginCreator could not find plugin Normalize_TRT version 1 namespace
E0612 20:28:44.454670 46 logging.cc:43] Cannot deserialize plugin Normalize_TRT
E0612 20:28:44.454697 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454715 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454736 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454742 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454763 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454777 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454795 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454813 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454831 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454838 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454857 46 logging.cc:43] getPluginCreator could not find plugin PriorBox_TRT version 1 namespace
E0612 20:28:44.454864 46 logging.cc:43] Cannot deserialize plugin PriorBox_TRT
E0612 20:28:44.454900 46 logging.cc:43] getPluginCreator could not find plugin NMS_TRT version 1 namespace
E0612 20:28:44.454907 46 logging.cc:43] Cannot deserialize plugin NMS_TRT
Note that I have a serialized plan file for SSD from running sampleSSD in TensorRT code.
Can anyone also provide a config.pbtxt for SSD?
from server.
I have implemented IPluginFactory with plugin and set
ICaffeParser* parser = createCaffeParser(); parser->setPluginFactoryExt(&pluginFactory);
and generate tensorrt model . when I employ it to server, it occur error
ERROR: Not a valid serialized plugin
And I also implemented plugin with IPluginV2 IPluginCreator. and generate lib.so
and use LD_PRELOAD=trtplugins.so but also occur the same error!
from server.
We're working on validating TRT plugin support in TRTIS. It appears to work in some cases but not all. We'll update here once we learn more.
from server.
We will be adding support for the default provided plugins shipped with TensorRT for now. We may add custom plugins support in the future if possible.
This means that plugins mentioned here will be supported by default in TRTIS in the future releases.
from server.
Support for default plugins as mentioned here is added on master and will be in 19.07. Please retry your model/s to let me know if it is not fixed for you.
from server.
I've successfully imported custom ops imported from onnx parser and registered there via LD_PRELOAD workaround.
from server.
Related Issues (20)
- Request for Improved Metrics and Real-Time Concurrency Reporting in Triton Inference Server
- Python AsyncIO infer does not support shared memory HOT 1
- client silent failure - E0422 05:03:24.145960 1 pb_stub.cc:402] An error occurred while trying to load GPU buffers in the Python backend stub: failed to copy data: invalid argument HOT 3
- CUDA Graph not work HOT 4
- [RFE] HandleGenerate equivalent for sagemaker_server.cc HOT 1
- The time spent on the inference request process far exceeds the model inference time. How can I determine where this additional time is being consumed?
- Casting NumPy string array to np_utils.Tensor disproportionately increases latency HOT 2
- On server/deploy/oci -> running "helm install example ." to deploy the Inference Server and pod doesn't get to running due to Liveness probe failed & Readiness probe failed HOT 1
- trt_profile_max_shapes not supported for ONNX-TRT backend HOT 1
- Failed to initialize Python stub + ModuleNotFoundError: No module named 'nvtabular', 'merlin' HOT 1
- does triton support different model-repository assemble into a batch? HOT 1
- Question: Which backends automatically warm up models? HOT 1
- [Question] Is it possible to shutdown Triton if we detect certain cuda errors ? HOT 1
- Perf Analyzer Error: Cannot send stop request without specifying a request_id HOT 1
- Python Backend: one model instance over multiple GPUs HOT 2
- Logs not getting generated with GRPC HOT 1
- Input data/shape validation HOT 7
- Manually update model repository index HOT 5
- Triton server execution is aborted in mac m3 pro as soon as a client sends a new request!!! HOT 2
- Unable to use triton client with shared memory in C++ (Jetpack 6 device) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from server.