How to build a message in a specific byte array? about capnproto-rust

[dwrensha]

serialize::new_reader() and MallocMessageBuilder are of course not what you want because they do allocations and must copy the underlying data.

On the reading side, you could manually read the segment table, construct a &[&[Word]] from the body of the message, and use that to make a SegmentArrayMessageReader. This is currently more cumbersome than it ought to be. We should have a FlatArrayMessageReader that takes care of this stuff.

The writing side is more problematic. I think that as things currently stand you actually cannot avoid copying the data. I had hoped that ScratchSpaceMallocMessageBuilder might be able to help, but unfortunately for our purposes here it zeroes the scratch space when it is dropped. To move forward on this front, I think we need a way to allow user-defined implementations of MessageBuilder to specify their own allocation schemes, as with the MessageBuilder.allocateSegment() virtual method in capnproto-c++.

[MarkSwanson]

Maybe instead of using a &[&[Word]] you could use a *const u64 and use pointer offsets.
This way you wouldn't have to create slices from raw pointers - and worry about their lifetimes.

Wrt the segment table, I see it partially defined like this:
(N * 4 bytes) The size of each segment, in words.

4 bytes per word = 32-bit word size. Rust defines it (incorrectly/incompatibly?) as u64:
pub struct Word {_unused_member : u64}

Part of the reason for using Capn'Proto is because it advertises zero-cost [de]serialization. I assume in C++ that I:

• create a builder that points to a mmap region
• set_x(), set_y(), ...
• I'm done writing, because the set_x() etc. methods have directly written to the mmap region.
• another process simply creates a reader that points to the mmap region and it's done; it can call get_x().

If my understanding is correct, is it the goal of capnproto-rust to get to this point?

• zero allocations, zero copies, no serialize::[new_reader,write_message] required?

Thanks!

(and thanks for all the great work so far)

[dwrensha]

Maybe instead of using a &[&[Word]] you could use a *const u64 and use pointer offsets.
This way you wouldn't have to create slices from raw pointers - and worry about their lifetimes.

I don't think it should be the library's job to do this conversion. How would it know what lifetimes to use?

Wrt the segment table, I see it partially defined like this:
(N * 4 bytes) The size of each segment, in words.

Maybe this would be more clear if it said "(N * 4 bytes) For each segment, an unsigned 32-bit integer representing the number of words in that segment."

If my understanding is correct, is it the goal of capnproto-rust to get to this point?
zero allocations, zero copies, no serialize::[new_reader,write_message] required?

Yes, we should be able to support such things. Perhaps you'd be interested in contributing to this? :)

[MarkSwanson]

I think it would be fun to work on this.
Currently swamped. Will circle back shortly.

[dwrensha]

Note that @ArtemGr has made some progress on this front: #25