[App]: hardware logging in multi-node setting about wandb

[nepfaff]

This would be useful to have

[thanos-wandb]

Hi @BramVanroy thank you for reporting this. May I please ask some more context, what's your current compute infra? and which ML frameworks are you mostly using?

[thanos-wandb]

Hi @BramVanroy just following up on this, to see if you could provide us with some additional information on your current multinode infrastructure so as to include those in a feature request for our engineers? thank you!

[BramVanroy]

Hi Thanos

I am running jobs on between 1 node, 1 GPU up to 10 nodes, 4 GPUs each. It seems to me that wandb does not correctly log hardware when it comes to multi-node settings.

[thanos-wandb]

Perfect, thank you @BramVanroy for the additional context. I was wondering what's reported in those runs, if you navigate in Files view and open the wandb-metadata.json file, under the "cpu_count" and "cpu_count_logical" entries. Does it not detect the correct hw info when it's multinode?

[BramVanroy]

Correct. It only reports the main node hardware configuration, but not the whole pool.

[thanos-wandb]

Great, thank you @BramVanroy for the clarification. I have logged this feature request with our engineers, and we will keep you updated here on any progress.

[abraham-leal]

For Nvidia GPUs, this may be a good workaround on this issue: https://wandb.ai/dimaduev/dcgm/reports/Monitoring-GPU-cluster-performance-with-NVIDIA-DCGM-Exporter-and-Weights-Biases--Vmlldzo0MDYxMTA1