cscfi / ansible-role-nhc

Install and configure Node Health Checker

ansible-role-nhc

Install Node Health Checker https://github.com/mej/nhc

Requirements

• A batch scheduler
• Internet access to Github

Role Variables

See defaults/main.yml for a complete list

If nhc_github is set to False it installs it from a pre-configured yum-repo. Default is True

nhc_github: True

If we set nhc_use_default_checks to True - then all the default checks in nhc.conf will not be added. Only the ones in nhc_checks list.

nhc_use_default_checks: True

nhc_checks:
 - { match: "*", name: "check_reboot_slurm", arguments: "" }
 - { match: "{gpu[1-19]}", name: "check_hw_ib", arguments: "40" }

nhc_rm: "slurm" # but can also be whatever else NHC supports like pbs or torque.

# setting nhc_rm to "" means we don't set the variable at all - autodetection
nhc_rm: ""

Dependencies

Example Playbook

Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too:

- hosts: servers
  roles:
     - { role: ansible-role-nhc }

License

MIT

Author Information

ansible-role-nhc's Issues

Use more variables

One would for example want to disable the infiniband speed check

Use NHC to ensuring that time is in sync on compute nodes

CSC internal tracking number: 208750

output of this command:

chronyc tracking|grep "Leap status"|cut -d: -f2|xargs

One could use "check_cmd_output" or "check_cmd_status"

nvidia GPU health checking

nvidia-smi should be enough

Add healthcheck scripts without modifying role

It would be nice if there were a way to add site-specific healthcheck scripts without having to modify the role.

What would be a good way to accomplish that? And if should work with ansible-pull too..

nvidia-smi --persistence-mode=1 to rc.local

GPU nodes needs to have added

nvidia-smi --compute-mode=3
nvidia-smi --persistence-mode=1

to /etc/rc.local

Otherwise SLURM start fails due to missing /dev/nvidia* devices. Having persistence-mode enabled would fix this issue.

dell hardware health status

• omreport tool might do this
  • would be nice if NHC can run this
  • makes this role depend on ansible-role-dell

Add option to install NHC from a yum repo

compare cuda installed and running libs

In FGI the healthcheck script checked nvidia-smi to see if userland and kernel modules still matched

since the module + libs can be updated via yum, but it requires a reboot

so the script drained+rebooted the node automatically

Allow using host match syntax for checks

On a heterogeneous cluster we need to check different things on different nodes.

E.g. some nodes have FDR Infiniband

{pe[1-64,gpu[20-22]} || check_hw_ib 56

while other nodes have QDR Infiniband

{gpu[1-19]} || check_hw_ib 40

Somehow it should be possible to generate configs like the above.