Comments (3)
Hi @ngessert, what you're seeing is the remnant of the worker that used to run on the machine - the last report has a timeout of 10 minutes (which is why it goes away after some time).
The question is why that worker did not shut down correctly (a proper shut-down should unregister the worker) - perhaps a full log of the autoscaler will help in understanding this (or the system log of the related cloud machine)
from clearml.
The last line of the EC2 instance's system log is just this (after instance was terminated):
[ 88.024534] cloud-init[1966]: + python -m clearml_agent --config-file /home/ec2-user/clearml.conf daemon --queue aws_m5xlarge
How does the unregister process work? Does the agent recognize somehow that the system where it is running is being shut down? Is it perhaps dependent on the OS? I am using Amazon Linux 2023.
from clearml.
@jkhenning any suggestion where I could start looking for what is going on with worker shutdown/de-registering? From a code point of view (the logs don't show anything)
from clearml.
Related Issues (20)
- Greater control in downsampling used in plots HOT 2
- Modify report_matplotlib_figure to Omit Iteration Value When iteration=None HOT 3
- Error when calling classes through Fire HOT 2
- Use a function of iterations (e.g. epochs) as the time scale for scalars and plots HOT 2
- Pathlib Path instances in a dataclass do not get tracked by task.connect() HOT 1
- delete datasets after call get_local_copy HOT 1
- report_matplotlib_figure of subplots HOT 2
- How to use Omegaconf without Hydra? HOT 1
- Scrolling log problem when using tqdm as training process bar HOT 5
- ClearML feature for integration KerasTuner is broken HOT 1
- Fix typo in docs and default sdk config HOT 1
- Executing clearml-task from cli with "-m" modules HOT 1
- Dynamic GPU/Queue Allocation for Workers in ClearML
- Add tag with Clearm-task (cli tools) HOT 1
- Problem creating datasets with Azure storage when multi file HOT 5
- Task creation failed!Always searching for this project? But I don't have it! HOT 1
- Support Megatron-LM training job on k8s cluster HOT 4
- Model.get_local_copy with specific download path. HOT 1
- "413 Request Entity Too Large" when uploading files to ClearML HOT 4
- legend titles broken in experiment comparison HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clearml.