Distributed training across multiple nodes about clearml

[bmartinn]

Hi @ebritsyn, not sure how this issue passed by without me noticing, apologies ;)
It's awesome to hear that you are using Horovod and Trains together!

Now back to your question:
Horovod sits on top of OpenMPI, and actually pipes all the stdout/stderr from all the nodes to the "main" (rank 0) node.

From your question I'm assuming this is a "manual" execution scenario (i.e. you launch the experiment training yourself). Do you have all the stdout/stderr from all the nodes on the "rank 0" experiment result page?

Did you add a specific code making sure you only create the experiment on the "rank 0" node, like #59 did?

Regrading source code & packaging, why don't you use the trains-agent to launch the code on all nodes, this way you can ensure the code & environment are exactly the same.

Basically all you nee to do is (assuming you have trains.conf already on your nodes):

$ pip install trains_agent
$ trains-agent execute --full-monitoring --id <your_task_id>

Or if you want to take the manual approach, you can a use trains-agent to build the experiment environment (packages and code) on every node, then execute the code manually:

$ pip install trains_agent
$ trains-agent build --id <your_task_id> --target-folder ~/my_horovod_experiment
$ cd ~/my_horovod_experiment
$ . bin/activate
$ python ./code/my_main_script.py