Add multipart filters about warp

[abonander]

Still, it might be useful for accepting form-data that doesn't include a file.

I deleted my original reply because I thought "duh, you can have optional file fields and thus get multipart request bodies that don't contain files" but if you're expecting a multipart body without any possible file fields then that's a lot of overhead for zero gain compared to urlencoding. Do you have a use-case in mind or is it more just supporting every possibility? If the client submitted a multipart body where urlencoding was expected it'd probably be better to emit an error unless you want to adhere to the "strict in what you send, liberal in what you accept" rule.

[abonander]

I looked through warp's docs when you announced it to see if integration with multipart was feasible but I found that the Filter trait is designed not to allow outside implementors since FilterBase isn't exported. Was this a deliberate design decision, and if so, is it a permanent one?

[seanmonstar]

Yes, Filter is a sealed trait for now, since I suspect there might need to be some breaking changes to its internal method (during development, I iterated a few times with passing different arguments to Filter::filter). New filters can be created by combining with map or and_then. If wanting to turn just any value into a filter, it can be done with warp::any().

For example:

let port = warp::any().map(|| env!("PORT"));

[abonander]

I see. I think multipart could provide a impl Filter helper wrapping filter::header::<ContentType>() and some hypothetical streaming body filter. The latter I think is all that really needs to be implemented for me to start experimenting on my end. I could use body::concat() if I really needed to, I guess.

[abonander]

Decode a multipart body via serde::Deserialize

I looked into this but serde isn't really designed for push- or poll-based interfaces like async APIs.

I pitched this idea on the Gotham-rs issue board about a derive-based solution to form processing that included validation, but I haven't really gotten anywhere with prototyping it yet.

[seanmonstar]

some hypothetical streaming body filter

I'm adding a warp::body::stream() filter right now :D

I looked into this but serde isn't really designed for push- or poll-based interfaces like async APIs.

Yea, I've also looked because I was curious asynchonously deserializing a stream of JSON. At least in serde_json, it kind of looks like it could be possible, with StreamingDeserializer, as the comments suggest one could try again if the deserializer returns an EOF error.

Still, it might be useful for accepting form-data that doesn't include a file.

[seanmonstar]

No, I don't have an exact use case, I was kind of assuming that if you used FormData in JavaScript, or <form>, it could use multipart/form-data without a file, but I don't have any proof.

[abonander]

Anything is possible, of course. You can specify enctype="multipart/form-data" without needing a file field and the docs for FormData say it encodes to multipart/form-data (I think URLSearchParams encodes to www-form-urlencoded).

But that's why instead of Serde, I was conceiving of using a custom derive designed for form data, since you could also include files in your form struct and the framework would just handle it. The idea was to eventually integrate with a renderer/template engine as well so the generated HTML form matches the expected structure. I just got sidetracked with work/life and forgot about it.

[abonander]

@seanmonstar what is that body stream filter gonna look like anyway? Are you refactoring filters to allow returning futures or are you making it work like a synchronous API?

[seanmonstar]

I've added it to master already. Using Filter::and_then, you can return a Future. So it could be done as:

let filter = warp::body::stream()
    .and_then(process_async);

It currently returns a pseudo impl Stream, as I expect the type to change slightly to allow adding in decompression filters in the chain as well.

[abonander]

@seanmonstar unfortunately I need the stream type to be nameable as the return type of my multipart filter would have to be impl Filter<Extract = (Multipart<ImplStream>,), Error = ...>

[seanmonstar]

If it's a filter built in to warp, it can both use the concrete type directly, and also not expose a generic.

[abonander]

I was hoping to start experimenting in multipart-async though since I've already got a very flexible API that's generic over the body stream. And impl Trait everywhere kind of hurts composability in general.

[seanmonstar]

I was hoping to start experimenting in multipart-async

Oh OK, as an immediate solution, it should be possible to turn it into a Box<Stream<Item = Box<Buf>>:

warp::body::stream()
    .map(|s| {
        Box::new(s.map(|buf| Box::new(buf) as Box<Buf + Send>))
            as Box<Stream<Item = Box<Buf + Send>, Error = warp::Error> + Send>
    })
    .and_then(|boxed_stream| {

    })

And impl Trait everywhere kind of hurts composability in general.

Yea, in some ways it might, but I've also been wanting to use warp to explore exactly this issue, if we can actually use impl Trait a bunch, or if it's only useful in tiny doses. This is relevant since async will be making everything use impl Future.

[abonander]

I guess it can be boxed trait objects, though I don't feel like that should be necessary. The types are constant, so the dynamic dispatch is purely for type erasure which feels like a code smell and won't perform optimally. I also designed my API over a splittable chunk type like bytes::Bytes so a boxed trait object will probably need copying to work.

I feel that impl Trait is more of an end-user convenience where some wrapper/combinator function around a library API produces an unnameable type due to closures or generators. In general in the library itself you're taking the closure or generator as an input so it can easily be covered by a type parameter.

[seanmonstar]

As part of v0.1.7, the types are now exported as BodyStream and StreamBuf.

[abonander]

@seanmonstar it's been a bit, kinda forgot about net stuff for a while.

Would it be feasible/efficient to implement split methods on StreamBuf like those on bytes::Bytes? Multipart-async is built entirely around those so it would have to perform a lot of copying otherwise.

[seanmonstar]

That should be doable.