💡[feat] the request to add a feature that releases resources automatically in case of a timeout or if the GPU utilization falls below a certain threshold about determined

[ioga]

hello,

we do not have this as a built-in feature. One of our engineers developed an unofficial script which can terminate a process if it's not utilizing GPU. you can integrate it into your CMDs to do what you want. doing that with interactive shells or notebooks would probably be a bit trickier.

[KyanChen]

Can you offer a implementation？

[KyanChen]

I have developed a python script.
master node runs: https://github.com/KyanChen/GPUClusterConfig/blob/dev/gpu_parser.py
worker node runs: https://github.com/KyanChen/GPUClusterConfig/blob/dev/gpu_monitor.py

[ioga]

I cannot offer a reference implementation besides the script I've already shared.
your code looks fine at a quick glance. it does not seem like it's going to handle the case when you have multiple GPUs per node. but hey, if it works for you, I see no problem with you using it.