Comments (4)
Please understand that at the moment we have no capacity to publicly release the inference node.
We can however give you some hints on what such an inference node would need to do:
- Load parameters
- path to exported TensorFlow model
- paths to one-hot-conversion files for input/label
- Setup node
- load TensorFlow model
- parse one-hot-conversion palettes
- create as many image subscribers as you have cameras
- synchronize image subscribers using
message_filters
and register callback function - create publisher for output image
- Callback/Inference Function
- create input tensors from image messages
- convert ROS image message to CV images using
cv_bridge
- resize images, if needed
- one-hot-encode images using the parsed palette
- convert ROS image message to CV images using
- run inference using TensorFlow C++ Library
- create output image message from output tensor
- one-hot-decode output tensor using the parsed palette
- convert tensor to ROS image
- publish output image message
- create input tensors from image messages
from cam2bev.
The 2Hz in the paper only relates to the sampling rate of the training sample generation process, i.e., training samples in our dataset are 0.5s apart.
We have a C++ ROS Node to run trained models in inference mode. I quickly measured its runtimes for the following configuration:
- NVIDIA GeForce RTX 3090
- uNetXST model
- 512x256 image size, 10 predicted semantic classes
- computation of the segmented images is not included in runtime
- single image: ~40ms
- 4 images: ~100ms
Possible areas of runtime improvement:
- The DeepLab models will likely run faster, especially DeepLab MobileNet. Their runtime also doesn't depend on the number of input images, since they work on a single homography image instead. In that case though one would also need to consider the runtime of computing the homography image.
- uNetXST can also be simplified by reducing its Encoder-Decoder-Depth (default: 5).
- Reducing image size will speed-up processing.
- Reducing the number of predicted semantic classes will speed-up processing.
Please also note that we have not focused on achieving minimal runtime. Typical model optimization techniques like pruning or quantization could further decrease latency.
from cam2bev.
Thank you. This was helpful. We will probably try to replicate this as first steps and then move to see if there are improvements. For our high speeds, a transformation to birds eye may be necessary as image stitching through homography alone seems to be unreliable.
from cam2bev.
Would you be willing to share the C++ ROS node with me? I'd like to do some experimenting and it would speed up the process to have your working node
from cam2bev.
Related Issues (20)
- how to generate this simulated data
- Training of deeplab-mobilenet and deeplab-xception failed HOT 2
- What changes to make to resume training from where it was left off? HOT 2
- Non-360 View HOT 10
- Frame rate HOT 2
- Training on original input HOT 2
- How to test on real-world images HOT 1
- Performance when evaluating custom data HOT 1
- BEV image HOT 8
- some issue about "homography_converter" HOT 10
- What should the regularization coefficient be set to? HOT 1
- drone camera config file HOT 2
- A few questions about your work HOT 1
- 0
- Training on Google Colab HOT 2
- Real-world application HOT 1
- field of view single-input model HOT 2
- the R matrix calculate order may have some problem? HOT 1
- Model for testing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cam2bev.