Explicitly execute multiple commands on the head node in parallel. about skypilot

[gmittal]

Another way to go about doing this is to sync file artifacts across nodes/locally, and then the user can either launch tensorboard locally or from a cpunode/gpunode. Services like tensorboard/jupyter should be launched interactively rather than as a part of the training task, imo.

[concretevitamin]

ParTask should be the best solution, if the base tasks are all single-node:

train = sky.Task(run='python train.py').set_resources(sky.Resources('V100'))
tensorboard = sky.Task(run='tensorboard --logdir .')
par_task = sky.ParTask([train, tensorboard]).set_resources(sky.Resources('V100'))

Any reason you wouldn't want to use this @Michaelvll?

Another way to go about doing this is to sync file artifacts across nodes/locally, and then the user can either launch tensorboard locally or from a cpunode/gpunode.

The default mode should be serving those remotely. Sync-to-local should be the optional mode. Event files/logs can be large and syncing to local is not always desirable.

Services like tensorboard/jupyter should be launched interactively rather than as a part of the training task, imo.

Agreed.

[concretevitamin]

BTW: This is a very good example to have!

[Michaelvll]

ParTask should be the best solution, if the base tasks are all single-node:

train = sky.Task(run='python train.py').set_resources(sky.Resources('V100'))
tensorboard = sky.Task(run='tensorboard --logdir .')
par_task = sky.ParTask([train, tensorboard]).set_resources(sky.Resources('V100'))

Any reason you wouldn't want to use this @Michaelvll?

Another way to go about doing this is to sync file artifacts across nodes/locally, and then the user can either launch tensorboard locally or from a cpunode/gpunode.

The default mode should be serving those remotely. Sync-to-local should be the optional mode. Event files/logs can be large and syncing to local is not always desirable.

Services like tensorboard/jupyter should be launched interactively rather than as a part of the training task, imo.

Agreed.

In my understanding, the ParTask may implicitly run the two tasks sequentially as it shows in multi-echo example when the resources are not enough. Will it guarantee the two task runs at the same time? Also, I found it not possible to specify the tensorboard to be run on the head node (not the workers) when we have num_node > 1 with ParTask.

[concretevitamin]

Will it guarantee the two task runs at the same time?

Yes, if the resource demands of both tasks are smaller than the VM resources. It is the same with Ray's semantics.

For example, train will use {V100:1, CPU:1} and tensorboard will use {CPU:1} (Ray semantics), while the ParTask grabbed a VM with n CPUs and {V100:1}. So they definitely should be run in parallel.

Also, I found it not possible to specify the tensorboard to be run on the head node (not the workers) when we have num_node > 1 with ParTask.

Actually, the ParTask backend doesn't support any base task with num_nodes > 1 at the moment. You should have run into an assert for that.