Zombie Workers with AWS Auto Scaler about clearml

[jkhenning]

Hi @ngessert, what you're seeing is the remnant of the worker that used to run on the machine - the last report has a timeout of 10 minutes (which is why it goes away after some time).
The question is why that worker did not shut down correctly (a proper shut-down should unregister the worker) - perhaps a full log of the autoscaler will help in understanding this (or the system log of the related cloud machine)

[ngessert]

The last line of the EC2 instance's system log is just this (after instance was terminated):
[ 88.024534] cloud-init[1966]: + python -m clearml_agent --config-file /home/ec2-user/clearml.conf daemon --queue aws_m5xlarge

How does the unregister process work? Does the agent recognize somehow that the system where it is running is being shut down? Is it perhaps dependent on the OS? I am using Amazon Linux 2023.

[ngessert]

@jkhenning any suggestion where I could start looking for what is going on with worker shutdown/de-registering? From a code point of view (the logs don't show anything)