Confusing argument layouts (send_receive_into) about rsmpi

[bsteinb]

The order of the function arguments is inherited from the original C bindings where you specify the buffer (message buffer or receive buffer) first, the communication partner (destination or source) second. This is true for both send and receive functions and since send_receive_into is just a pair of send and receive_into operations packed into a single function, its argument list is the concatenation of the argument lists of those two functions. I am reluctant to give up this similarity to the C bindings.

Also, note that your reading of the improved argument list is not in fact what send_receive_into does. It performs two operations on the calling process: it sends msg to destination and receives a message from source into buffer with the added value that these two operations will not cause each other to block, e.g. source and destination could both refer to the calling process itself and not cause a deadlock.

[flukejones]

Ah! I thought that might be the case, thank you. And thank you for the clarification - I've got a fair bit to learn yet 😄

This isn't the most appropriate avenue to ask questions, but: do you have an example of gathering a nested vector at all?

I so far have a root process sending out the required data via a for p in world.size() loop to each process, where each process runs a mirror for loop to receive the nested vector and push to another vector - using send_receive_into for the root process and receive_into for the non-roots.

It seems I need to do the inverse with each non-root iterating through the Vec<Vec<_>> and using a view to send the data, where the root process... receives_into? in a for loop checking the rank of the msg?

I can't quite see how use of gather_into_root works here - this would concatenate each sent vector in to one contiguous vector right?

[flukejones]

Hmm, another step of my confusion here was

            for c in start..end {
                unsafe { // scoped to drop the mut ref so that the column can be moved
                    let msg_to =
                        View::with_count_and_datatype(&self.current[c][..], 1, &data_type);
                    // the process to send to
                    // This is the inverse of how you would expect it to operate;
                    // eg root.send(&msg_to, &process)
                    process.send(&msg_to);
                }
            } // snippet from my Game of Life code

Where process.send is;

Send the contents of a Buffer to the Destination &self
I didn't realise that meant literally calling send() on the process you wanted to receive the data. Essentially sending the data to itself.

Followed by root.receive_into(&mut into); for example where I was reading this as "root receives this in to its buffer" - as opposed to "in the calling process we will receive from root and put it here".

I can get some time in the next while (I have assignments and exams coming up) I'll see if I can draft some sort of introduction or tutorial for you you use - it might be the ideal thing for me to write since I'm an outsider to the codebase and have likely hit many of the same roadblocks other newcomers will.

[bsteinb]

I can't quite see how use of gather_into_root works here - this would concatenate each sent vector in to one contiguous vector right?

Yes. MPI knows nothing about Rust datatypes so its operations are defined on C arrays. In this situation you can either go with your solution of emulating the gather operation with a series of send and receives into the individual sub-vectors or gather into a flat buffer and then partition it afterwards. The second solution might give better efficiency at least for the inter-process communication of the data.

[bsteinb]

I didn't realise that meant literally calling send() on the process you wanted to receive the data. Essentially sending the data to itself. Followed by root.receive_into(&mut into); for example where I was reading this as "root receives this in to its buffer" - as opposed to "in the calling process we will receive from root and put it here".

I can get some time in the next while (I have assignments and exams coming up) I'll see if I can draft some sort of introduction or tutorial for you you use - it might be the ideal thing for me to write since I'm an outsider to the codebase and have likely hit many of the same roadblocks other newcomers will.

Yeah, this might be a good idea, eventually. The documentation at the moment is geared towards people who have already used MPI through its C or Fortran bindings and mostly serves to build the connection to those. I can imagine that someone who is completely new to the API might struggle at first. However, since rsmpi is not quite finished yet, I don't know whether it makes sense to write long form documentation just now. It would be another place that has to be modified in case there are API changes, and mostly, I think, without the help of a compiler that warns you when things get out of sync.

As for your observations on the &self argument of send and receive functions: I think this is also something that would be less surprising to someone who knows the original C bindings. The "executor" of MPI point-to-point operations is not explicitly specified, it is always implicitly the calling process.

In the C bindings, receive has the signature receive(buffer, source) (with buffer and source actually comprising multiple actual arguments). I translated these into a receive method on a value that implements Source. I could have translated it as a free function as well. I am not sure why I made the choice initially. Maybe I thought it was a better fit for a Rust library. Maybe I started with send and thought method call syntax would be appropriate due to the calling a method <--> sending a message analogy from classical OOP.