Multi-Node GPU train Example. about climax

[rejuvyesh]

Multi-node experiments setup usually depends on your local cluster setup. I would recommend following the PyTorch Lightning documentation at https://pytorch-lightning.readthedocs.io/en/stable/clouds/cluster.html. Otherwise, the current code is already setup for doing multi-node experiments. And would just require setting the environment variable NODES to the correct number and setting --trainer.num_nodes to the correct number.

[mahm1846]

Thanks for the reply.

1. By "Multi-node", I was referring to frameworks such as Horovod or Colossal, but Pytorch DDP is alright, too.

2. I run into another problem with the conda install. Our cluster run on Centos 8.5, which only has upto "glibc-2.28". I followed the conda install instructions without any error. But there is a runtime error, "torchdata" is looking for "glibc-2.29".
   Does Microsoft has any conda build or do you know any build channel that uses "glibc-2.28" for Pytorch products?
   or, Can you suggest any workaround ?

I look forward to your reply.

[rejuvyesh]

torchdata is easy to expunge from the current code (although you could also just use docker instead). At the moment it's only being used to get a list of files: dp.iter.FileLister. You can replace that with your own function that does the file listing. Something like:

def get_files_from_root(root):
    if os.path.isfile(root):
        path = root
        yield path
    else:
        for path, dirs, files in os.walk(root):
            files.sort()
            for f in files:
                yield os.path.join(path, f)

            dirs.sort()