Comments (4)
Makes total sense @cthorey - Would you care to issue a PR?
from clearml.
I though about it but then I realize, we have still no way to guarantee that the agent does not pick up a new task while the instance is taken down by the cloud provider. What would be better would be to be able to detect when the agent have been taken down and reschedule Task that have been interrupted this way.
I raise an issue allegroai/clearml-agent#188 here which prevents this for now.
Specifically, when an instance is taken down, SIGTERM are sent to running processed and the running task are marked as completed. What would be better would be to mark them as fail so that we have the option to reschedule them via the retry_on_failure parameter which we can pass to the PipelineController.
from clearml.
Sounds like we're mixing up a number of points @cthorey.
- Your original post - A race condition where an instances activity status is obsolete by the time the autoscaler takes action for taking it down.
- The status of a task once its executing agent was explicitly terminated (which you address in clearml-agent#188) and its effect on pipeline logic.
These should probably be handled independently. WDYT?
from clearml.
Yep - I agree they should be handled independently.
Regarding 1. and hence this issue, I think we can reasonably close it given that, as I said above, we have no way to guarantee that the agent does not pick up a new task while the instance is taken down by the cloud provider.
from clearml.
Related Issues (20)
- Registering models from lightning not working (different than pytorch-lightning) HOT 2
- GPU monitoring failed getting GPU reading, switching off GPU monitoring HOT 6
- async variant of get_mutable_local_copy HOT 2
- Light theme for the dashboard HOT 1
- Plot comparison in a single figure not working for plots other than barplots HOT 3
- API calls fail for model with deleted parent task. HOT 1
- Scalar logging bug with Fire HOT 6
- multilanguage support HOT 1
- is it possible to upload bytes object instead files? HOT 1
- Add retries and chunking parameters to upload_folder HOT 2
- Add_external_files for Hyperdatsets HOT 1
- Add Progress Bar for downloading (and maybe uploading) artifacts HOT 2
- [ERROR] [CLEARML.auth] Error getting token - FileServer HOT 1
- Many ClearML Dataset functions and parameters not available for Hyperdatasets, also it would be clearer to change name of allegroai Dataset to Hyperdataset HOT 1
- How to generate api services content?
- Scalars, Plots, and Debug Samples hang and stop logging with large git diff and Tensorboard Images HOT 3
- Relative path is incorrect when using `Dataset.add_external_files` for local file without the `file://` prefix HOT 1
- RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock. HOT 1
- When attempting to send a request with event size exceeding max_req_size, several subsequent events are also lost. HOT 1
- Only grouped metrics are displayed in the WebUI HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clearml.