Plan to support more than 2GB? about microsoft.io.recyclablememorystream

[benmwatson]

This is something that has come up every so often, and I'm thinking about it. One of the issues would be taking this farther from the underlying MemoryStream interface that we try to adhere to, but I think it's reasonable to consider moving beyond it.

[benmwatson]

@pengchen0692 @paulomorgado I have submitted a PR to open the size limits. I'm curious what you think.

[benmwatson]

Just an update on this: I am working on code changes for this request, as time allows. It isn't a trivial change, unfortunately, and will constitute a breaking change in the library (ints changing to longs in a few public places). There are also a lot of unit tests that need to be added.

I also have to decide how to handle things like multi-GB streams. .NET still has maximum array sizes. So even if you can have a stream that is 1 TB in length (with thousands of blocks), you can't call GetBuffer on that. Lots of little details like that.

[paulomorgado]

A way to not introduce breaking changes is to introduce overloads for both int and long, where possible.

It the stream length goes beyond the maximum array size, GetBuffer() should throw an InvalidOperationException.

But I don't know if managed memory is the best option for more than 2GiB. At that point, it might be a good option to go with unmanaged memory.

Unmanaged memory might be a good option for the pools of RMS even from smaller sizes.

[benmwatson]

At the beginning, I was considering adding overloads. I think I assumed it would just be too messy, but it probably didn't end up being so. I'll rethink this.

I'll change GetBuffer.

I think unmanaged buffers might be a bridge too far at this point. That would dramatically change the implementation, and could potentially restrict the users of the library, but I'll keep that in mind.

[paulomorgado]

Unmanaged buffers would, definitely, be a whole new implementation.

We might get away with the same implementation for the stream but with a different manager.