GPU questions about calamari

[ChWick]

• we never tested training on several GPUs, however, Tensorflow does support multi-GPU training, thus this might work out of the box. Possibly, a high batch size is required.
• The charts in your link suggest using 16-bit floats for the new RTX cards. However, Calamari uses 32-bit floats. We never tested 16-bit floats for training, and the effect on the accuracies. Thus, I cannot say if the RTX2070 is significantly faster than the older 10XX generations.

@ChWick

• In-order to train using multi-gpu, what would be the training command?

• Also, can Calamari handle 5 Terabytes of training data? ( Not joking )

• What should my settings be when dealing with Terabytes of training data?

[ChWick]

• By default Tensorflow aquires all available Resources, therefore not additional commands should be required (check by nvidia-smi)
• Yes and no. Currently, calamari loads all data into the RAM before training, and streams it into tensorflow. If you have 5 TB of RAM this should work (joking). So, no, it will not work. However, one could consider to move some logic to the data streaming pipeline to load/unload/preprocess the data on-the-fly. Another problem is, that calamari automatically computes the codec by loading all text files and computing the set of all characters. This will take a long time for 5 TB of data, which is why the codec should be given by hand. Unfortunately, both are larger changes.
• Thus, you can currently only train on a small amount of data.

@ChWick Why didn't you use Mixed-Precision? It:

• Halves the memory requirements.
• Speed-up the training substantially.
• No loss in model accuracy.
• No need to modify hyperparameters.
  https://devblogs.nvidia.com/mixed-precision-training-deep-neural-networks/
  https://arxiv.org/abs/1710.03740

https://www.youtube.com/watch?v=i1fIBtdhjIg

@ChWick
Will Calamari be able to synchronize the batch normalization among the 2 GPUs?

So, Calamari training speed would scale by a factor of 2x using 2 GPUs?
What would be the most important factor in Calamari training: Bandwidth, Flops, or number of GPUs?

@ChWick Thanks man for the on-the-fly branch! 😎

Note that you would need Tensor Cores to fully benefit from half-precision. Also note that the gtx 10x series are mostly optimized for fp32.

• NVIDIA/apex#76
• https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_20_series