Comments (6)
Hi,this probably can done but will likely require adjusting the code. We've never tried training on multiple GPUs.
from os2d.
Okay. In my case, I have a 48GB GPU, but the training occupies only 8GB, thereby taking three days to train the detector. How were you able to do this in a shorter time?
from os2d.
But what is the processing load of your GPU? if it is low you can try increasing batch size.
Another idea: it might be fine to train for significantly fewer iterations, you just need to monitor the behaviour of the validation loss to stop the process early.
from os2d.
GPU runs at 100% capacity, but most memory is left idle. Increasing the batch size in the config file isn't reflected in the final parameters. Did you face this issue while experimenting?
For the second approach, the current code doesn't have a tfboard support. Should I monitor the logs instead?
from os2d.
GPU runs at 100% capacity, but most memory is left idle. Increasing the batch size in the config file isn't reflected in the final parameters. Did you face this issue while experimenting?
It sounds weird, changing train.batch_size
and train.class_batch_size
should definitely change the training process - at least the GPU memory usage should go up.
However, if the GPU is already at 100% simply changing batch size is not likely to increase training speed.
For the second approach, the current code doesn't have a tfboard support. Should I monitor the logs instead?
Yes, the code does not have tensorboard but it includes another visualization tool: os2d/utils/plot_visdom.py
from os2d.
Got it, thanks!
from os2d.
Related Issues (20)
- Same class images with different class_ids get different results? HOT 1
- Attributes of grozi.csv HOT 3
- Error on using multi GPU setting HOT 2
- What is 3264 in the src? HOT 2
- Cannot find 'net' in the checkpoint file HOT 2
- Need for OrderedDict HOT 2
- CUDA out of memory HOT 4
- Security alert from dependabot HOT 3
- Segmentation fault HOT 4
- Error in visualisation HOT 1
- New form of evaluation HOT 3
- some new ideas about YOLO to os2d HOT 1
- Memory Leak when training and long evaluation time HOT 3
- Long Evaluation time and question about environment for training HOT 4
- Dataset scales when training with custom data HOT 2
- Evaluation Dataset HOT 1
- Unable to reproduce correct results from demo.ipynb HOT 2
- read image with cv2.imread
- how can I train my custom dataset base on OS2D V2-train model?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from os2d.