Comments (13)
Certainly, though as I said it's a bit of a mess at the moment. I will tidy it up, I just needed to prove it can work before moving forward. Also worth noting I'm not sure if the onnx file would work on GPU, not tried yet.
hack to ms_deform_attn so it doesn't use the custom op (and why it might not work on gpu)
Changes to oneformer model to handle missing task in input dict, and only output one task type
script to export the onnx file
Export command would look something like
python ./demo/export_model.py --config-file ./configs/ade20k/convnext/oneformer_convnext_large_bs16_160k_2gpu.yaml \
--format onnx \
--export-method tracing \
--sample-image ../datasets/512x512.jpg \
--run-eval \
--output ./output \
MODEL.IS_TRAIN False MODEL.IS_DEMO True MODEL.WEIGHTS ./output/ade20k_convnext_large/model_0099999.pth
Currently the dimensions of the sample image need to match the dimensions the model was trained at.
I think the rest of that commit it just noise (I was having odd issues with pybind11) and a few of my own bash scripts for convenience.
If it works for you please let me know and maybe we could work on a cleaner approach to add official support to OneFormer.
Tom
PS
Remember from my post above you have to use pytorch 1.12.1, torchvision 0.13.1
from oneformer.
i've managed to export the model to onnx, there's a few issues at the moment to get it working
- I wasn't sure how to get MSDeformAttnFunction to export (Think it's possible) so commented it out and used the ms_deform_attn_core_pytorch fallback
- Onnx expects just "image" in the input dictionary, there was no way to pass "task" so I just did a check if it was missing and added one to the dict so it didn't crash
- Onnx expects only a list of tensors as output so as I was only interested in semantic I just output that
- There's an op used, can't remember which, that pytorch 1.10 onnx export didn't support. You need at least opset 16, so I had to switch to pytorch 1.12.1, torchvision 0.13.1, then build detectron2 from source as couldn't seem to find a compatible prebuilt.
With those changes the export_model.py script worked with a few small changes and I was able to use the model in onnxruntime.
from oneformer.
I'm not sure if the team has any ready solution you could use, but if not, I have a backup suggestion for you. OneFormer is built with Detectron2. I deployed a similar segmentation model in the past, and I used this script to convert, and it worked.
from oneformer.
Hi @abhigoku10, thanks for your interest in our work. We don't have an already written script to convert the model to TensorRT or ONNX format.
I think the conversion should be pretty straightforward. You can try following the script shared by @SkalskiP (thanks for sharing!). If you need any assistance from our side, please let us know.
from oneformer.
Feel free to re-open the issue if you need any help.
from oneformer.
Hi @tomhog, could you please share the changes you did for exporting to onnx? Thanks a ton!
from oneformer.
@tomhog have you tested gpu support yet?
I was hoping to use OneFormer as an onnx model, but not having GPU support would mean, that I have to use a different model or implement my own.
from oneformer.
I'm very new to ML, so I apologize if my question is naive. Is it possible to convert this model to PyTorch Lite? I'm curious to see if it's possible to use this model with https://playtorch.dev/.
I was hoping to follow this: https://playtorch.dev/docs/tutorials/prepare-custom-model/
But I don't think it's that straightforward. I would appreciate any guidance if possible! Thank you.
from oneformer.
Hi @AAAstorga, thanks for your interest in our work.
You should follow the tutorial on using DETR with PlayTorch, as OneFormer and DETR are both built using detectron2.
https://playtorch.dev/docs/tutorials/snacks/object-detection/.
Also, it might be better to create a new issue for PlayTorch.
from oneformer.
Thank you @praeclarumjj3 - I appreciate the response. Do you have any directions on how to load the pretrained model in Python with PyTorch so I can convert it to a mobile friendly version? This might sound like a simple question, but I'm just starting to learn about all of this. Thank you!
from oneformer.
You can refer PyTorch tutorials: https://pytorch.org/tutorials/
from oneformer.
@tomhog I'm using your code to convert onnx and I'm having this problem is there any way to fix it look forward to your reply!
Traceback (most recent call last): File "./demo/export_model.py", line 249, in <module> exported_model = export_tracing(torch_model, sample_inputs) File "./demo/export_model.py", line 154, in export_tracing torch.onnx.export(traceable_model, (image,), f, verbose=True, opset_version=16, do_constant_folding=False, input_names=["input"], output_names=["output"], dynamic_axes={})#STABLE_ONNX_OPSET_VERSION) File "/home/lbm/lbm_src/conda_env/env/oneformer/lib/python3.8/site-packages/torch/onnx/__init__.py", line 350, in export return utils.export( File "/home/lbm/lbm_src/conda_env/env/oneformer/lib/python3.8/site-packages/torch/onnx/utils.py", line 163, in export _export( File "/home/lbm/lbm_src/conda_env/env/oneformer/lib/python3.8/site-packages/torch/onnx/utils.py", line 1110, in _export ) = graph._export_onnx( # type: ignore[attr-defined] RuntimeError: ONNX export failed: Couldn't export Python operator NATTEN2DQKRPBFunction
from oneformer.
Certainly, though as I said it's a bit of a mess at the moment. I will tidy it up, I just needed to prove it can work before moving forward. Also worth noting I'm not sure if the onnx file would work on GPU, not tried yet.
hack to ms_deform_attn so it doesn't use the custom op (and why it might not work on gpu)
Changes to oneformer model to handle missing task in input dict, and only output one task type
script to export the onnx file
Export command would look something like
python ./demo/export_model.py --config-file ./configs/ade20k/convnext/oneformer_convnext_large_bs16_160k_2gpu.yaml \ --format onnx \ --export-method tracing \ --sample-image ../datasets/512x512.jpg \ --run-eval \ --output ./output \ MODEL.IS_TRAIN False MODEL.IS_DEMO True MODEL.WEIGHTS ./output/ade20k_convnext_large/model_0099999.pth
Currently the dimensions of the sample image need to match the dimensions the model was trained at.
I think the rest of that commit it just noise (I was having odd issues with pybind11) and a few of my own bash scripts for convenience.
If it works for you please let me know and maybe we could work on a cleaner approach to add official support to OneFormer.
Tom PS Remember from my post above you have to use pytorch 1.12.1, torchvision 0.13.1
@tomhog Hi, sorry to bother you. Have you ever encountered the problem of significant drop in accuracy when using the onnx model to predict? I compared the output with the same input, the difference is in MSDeformAttnFunction.apply and ms_deform_attn_core_pytorch.
Picture 1 below is a comparison of the output of onnx using cuda and the original model.
Picture 2 below is a comparison of the output of onnx when using CPU and the original model.
Obviously the accuracy is slightly worse when using CPU, but it is directly 0 when using CUDA. Have you encountered this problem? Or can you provide some suggestions?
from oneformer.
Related Issues (20)
- undefined symbol: _ZNK3c107SymBool10guard_boolEPKcl
- whether I need to download Swin transformer pretraining weight for training
- Swin large backbone warning: "norm.bias will not be loaded. Please double check and see if this is desired." HOT 1
- AssertionError when trying to reproduce result HOT 2
- Inference time
- Class labels Issue
- Different result in different GPU
- Update colab demo HOT 1
- Resources required to fine-tune this model with swin-l HOT 13
- Potential code bug but always good model
- Freezing layers for DiNAT model
- Cityscapes val SOTA results not found in the paper HOT 2
- Training ADE20K for Instance Segmentation Only
- diffrent result on HF and my PC HOT 3
- Wrong citation
- What is the expected dimension sizes for the outputs dictionary from sem_seg_head? HOT 3
- Installation and setting up this repo is challenging HOT 1
- How to set prefetch_factor? HOT 1
- How to set the image size to run demo ?
- Runtime Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oneformer.