craftspider / department Goto Github PK

Currently, release 0.3 is waiting on a couple things, hopefully:

• JustMetadata<T>: Allows implementing CoerceUnsized on most handle types - rust-lang/rust#97052
• GAT Stabilization: Allows using GATs on stable - rust-lang/rust#96709
• Optionally, something to allow Handle<T, for<T> This<T> = Self::Handle<T>> to be added as a bound to handles

Currently, GATs and Pointer Metadata are the two biggest nightly features we need.

GATs are planned to stabilize in the not-too-distant future. This is a major step towards the crate compiling on stable, and will allow removing the GAT feature. Would like to have them stable before the next big release.

Relevant PR: rust-lang/rust#96709

Pointer Metadata has proposals to stabilize some types, but looks to still be a while off. This won't block the next release, but I intend to keep an eye on any movement in this area.

Oh hi! I hadn't seen this until I saw it pop up in rust-lang/rust#97052.

I'm part of alloc-wg and about two months ago I picked up the Storage API again in this repo: https://github.com/CAD97/storages-api (docs: https://cad97.github.io/storages-api/storage_api/index.html)

As part of this refinement and simplification push, I ended up converting the Storage trait to be untyped: that is, Storage::Handle is no longer a GAT, and notably this means that handle resolution can return regular references to [MaybeUninit<u8>] rather than pointers with difficult to explain invalidation rules.

There's some miscellaneous renaming involved, but also the simplification brought us down to a set of just four traits: Storage, SharedMutabilityStorage, PinningStorage, and MultipleStorage. I've been advised (and agree) that controlling complexity is important here for a path to std inclusion; even with this simplified API surface without GAT, it's still a contender for the most complex API exposed by std. And that complexity is safety critical.

I'm fairly convinced that the current shape in my repo is minimal for what we want to support, but I wasn't thinking the previous POC was terribly over-engineered¹ either, so it's possible that a further simplification is hiding.

matthieu-m agrees that the untyping of storage is probably a good direction; the previous storage API was not typed due to some conscious decision due to necessity, but because it evolved from C++'s std::allocator design, which is typed, and matthiue-m just hadn't considered using an untyped interface. This does mean that usage of the storage traits becomes more unsafe/difficult, but this is easily addressed by adding a typed RawBox around Storage, RawVec around Storage of slices, and maybe RawArena around MultipleStorage? as a bridge between raw storage and high-level collections.

(It's also quite possible that matthieu-m overlooked the possibility of making handles untyped because of the relative newness of the pointee metadata APIs at the time, in order to draw the conclusion that S::Handle<T> can be replaced by (S::Handle, <T as Pointee>::Metadata).)

Anyway, getting to the point:

I received a Rust Foundation Community Grant. My project is sort of two projects; the first is unrelated work in the tracing ecosystem, but the second half (scheduled Oct..=Dec) is to develop and push the Storage API to an MCP. rust-lang/rust#97052 is actually an early start on prerequisite work for upstreaming storages, as otherwise a storage-generic Box can't be coerced.

pub struct RawBox<T: ?Sized, S: Storage> {
    handle: S::Handle,
    metadata: <T as Pointee>::Metadata, // -> JustMetadata<T>
    storage: S,
}

My plan for MCPing the Storage API is basically broken down into the following three parallel steps:

• Recruit a T-lang member to liason/proxy/sponsor/whatever-the-word-is the proposal.
• Provide a proof of concept implementation for minimally all currently allocator-generic std collections on top of the Storage API.
  • I'm still up-in-the-air about whether to do this in-tree on the std impls directly in a fork, or to reimplement them in a separate repo (like you've done here).
  • Part of what makes this interesting is that this makes std collections #[no_std] compatible, so while refactoring in-place has the benefits of full API coverage and ease of "build with my fork of std" testing, wanting to move things into libcore means the diff would be little better than an external crate. Plus having to deal with std iteration times.
• Write the MCP.
  • Document the supported use cases / design constraints / etc.
  • Document why the API has the shape it does.
    • My intent here is to guide the reader through inventing the storage traits themselves, like the "reinventing tower::Service" article.
  • Make the argument for putting this into std in addition to the current unstable Allocator trait.

Ultimately, I think that this MCP is where the Storage API either sinks or swims. Either it gets accepted for in-tree implementation/experimentation, or it gets rejected, likely on the matter of being insufficiently motivated for std inclusion given its complexity. Especially given the async-wg group w.r.t. async in traits (e.g. dyn*), now really does feel like the time to say if we want to make Box the "owning handle to some storage" or if it's just an owning pointer to allocated memory. My claim is that Box<T, ?> can be both &move T and dyn* T, as well as InlineMaxSizeDst<T> and so much more, so its a matter of proving it worth it.

I'd like to work together, or at the very least avoid duplicating work you've already done unnecessarily. Adding your experience working with (your modified copy of) the storage API will help to strengthen the motivation provided in the MCP.

...I don't really know how to end this info dump. I guess...

TL;DR: Hi 👋 I'm working on the Storage API and would like to incorporate your experience working with it.

Currently, only a small number of handles can be unsized. Preferably, all handles should be coerce-able. This requires some Rust features though:

• JustMetadata<T>: Allows MetaHandle and OffsetMetaHandle to be unsized - rust-lang/rust#97052
• Fallback handle requires unsizing enums. This may be possible to define / implement, just needs a motivating use

This is currently blocked on a lack of for<T> or similar style of bound. The metadata and addr methods work fine generically, but anything using casts immediately runs into unproveable bounds, so it won't be added as a bound until those can be resolved somehow.