Queries on OpenSHMEM collectives usage about grid

[paboyle]

yes... it is incorrect at present. Thanks.

Hasn't been tested for some time, and fails under OpenSHMEM, and yo may have saved me a lot of time debugging this.

Also -- perhaps you can help me. I'm really annoyed that the OpenSHMEM and CraySHMEM argument ordering for shmem_align is reversed. Any comments ?

Peter

[naveen-rn]

I'm really annoyed that the OpenSHMEM and CraySHMEM argument ordering for shmem_align is reversed.

It was a Bug, fixed now. Fix will be available from CraySHMEM/7.5.1

This Bug went unnoticed because, shmem_align is one those least used routines in CraySHMEM and there are some fundamental functional differences between OpenSHMEM and CraySHMEM on this routine. By default, in CraySHMEM the maximum alignment value allowed on shmem_align routine is 64 bytes. And, we don't prefer users to self align anything more than 64 bytes and attempting anything more would error out. This is because, supporting alignment values greater than 64 bytes would create too much memory wastage.

That said, if there are any actual use cases which shows some performance benefits for alignments greater than 64 bytes, we can always look for ways to implement it. Let me know your shmem_align usage, I can look at it.

[paboyle]

The align is normally the L2 line size, which on Intel is 64Bytes, but I would prefer to support
128B on Power for example.

See no need to go to a page size for alignment, though, despite prefetches not crossing page boundaries. Rather, page sizes and cache line sizes should both go up after around 45 years....

I honestly think Intel should move to a larger L2 line size as L2 prefetch
overhead gets suppressed by the line size. Issuing L2 prefetch, L1 prefetch and load for each individual 512 bit vector is now ridiculous.

Clearly if the vectors went to 1024 bits, they and you would need >128B align, but... surely the above argument about line touching is true.

Or to put this in more "sexy" Hennessy and Patterson language, there is no way to obtain gain from spatial locality of reference in the memory system when the vector size is equal to the cache line size. :)

[naveen-rn]

That said, if there are any actual use cases which shows some performance benefits for alignments greater than 64 bytes, we can always look for ways to implement it. Let me know your shmem_align usage, I can look at it.

FYI, shmem_align() in Cray SHMEM is fixed. We can use any value for alignment size.

[paboyle]

Just a small comment -- Chris Kelly cleaned up the SHMEM comms recently and made it all work again. Closing thanks.