bug with mft.reshape about mesh

[EiffL]

Hey @b-remy, I opened a new branch based on a different version of horovod, and so far it looks like I'm not getting deadlocks.... but this different horovod impl needs a bit more testing.

Can you work on the transpose issues even with this bug and deadlocks happening? Or do we need to solve the deadlocks first?

[EiffL]

so, @b-remy I think this solved itself for some reason when switching to max_vd, right? can we close it?

[b-remy]

On a simpler example mtf.transpose also has an issue:

Comparing tf.transposeand mtf.transpose on a 4x1 mesh:

There is no deadlock for this one, but we can clearly see that the transpose is correct only on one slice of the cube.

The functions from hvd_simd_mesh_impl.py that mtf.transpose calls are: allconcat and slicewise and slice.

[b-remy]

Yes we can close it :)
It was indeed solved when using the new version of horovod and switching to the max_hvd branch.