Synchronous Training about graph-learn

[archwalker]

Initialize optimizer with use_locking=True flag in DistTFTrainer will perform synchronous training in distributed settings. Modify code here

[baoleai]

We use GraphSAGE as an example to show how to train in parallel on multiple machines. Currently we just provides asynchronous training example, you can simply replace it with synchronous training by using synchronous optimizer in TensorFlow.

[power1628]

Initialize optimizer with use_locking=True flag in DistTFTrainer will perform synchronous training in distributed settings. Modify code here

Are you sure?
The use_locking only guarantees multi-thread safety, not distributed synchronous training.
BTW, I don't think this issue should be closed.

[archwalker]

Initialize optimizer with use_locking=True flag in DistTFTrainer will perform synchronous training in distributed settings. Modify code here

Are you sure?
The use_locking only guarantees multi-thread safety, not distributed synchronous training.
BTW, I don't think this issue should be closed.

I misunderstood you. Basically, to use distributed synchronous training, one needs to:

1. Wrap optimizer with tf.train.SyncReplicasOptimizer here
2. Add a synchronous hook in MonitoredTrainSession here

A synchronous example will be posted soon.

[power1628]

This issue should be re-opened since it's not fixed.

[baoleai]

The question is how to synchronize training using TensorFlow, the above discussion has given advice.