trying distributed multigpu about colbert

[okhat]

Hi Minsoo!

This is a sample command for v0.2 multi-GPU training with 2 GPUs:

CUDA_VISIBLE_DEVICES="0,1" python -m torch.distributed.launch --nproc_per_node=2 -m colbert.train \
  --amp --doc_maxlen 180 --mask-punctuation --bsize 32 --accum 1 \
  --triples /path/to/MSMARCO/triples.train.small.tsv \
  --root /root/to/experiments/ --experiment MSMARCO-psg \
  --similarity l2 --run msmarco.psg.l2

It seems like the code was handing with your command due to --local_rank. This will be passed automatically; don't worry about it.

Happy to help with any other issues you may face. Will keep this issue open unless you'd like to close it.

[mnskim]

Hi Omar,
The command works well without any hang, thank you!!