Access to the HTTP trailer about fetch

[mnot]

Trailers are used extensively inside of back-end networks and CDNs. Server-side folks generally are very keen to get them supported because they help deal with pain around buffering, etc. E.g., Varnish would LOVE to be able to send ETag in trailers.

The problem has always been browser support. The use cases might not be compelling to browsers, and browsers might have legitimate concerns about buffering, code complexity, etc., but saying that there aren't any compelling use cases because browsers haven't done it yet is pretty presumptuous.

[martinthomson]

Note that trailers are very poorly supported in HTTP implementations. Microsoft have explicitly said that they ignore them; others do similar things.

I think that the right thing to do now is to give up on trailers.

[annevk]

We discussed this at the HTTP workshop:

1. It's okay to expose trailer headers sans semantics in the API and browsers are okay with that.
2. HTTP trailer semantics need to be clarified httpwg/http-core#16 and clearly separated from HTTP headers. At that point implementing semantics on a per-header basis might be reasonable and browsers would consider it.

[annevk]

Everyone subscribed to this issue might be interested in #473. I wrote a test for trailer() and filed bugs against the major browsers to support it. If anyone wants to write more tests that'd be appreciated.

[tyoshino]

Yes, we need to discuss that point. /cc @domenic

Like the headers being atomic, my gut feeling is that the trailers should also be handled without backpressure once the body has been consumed. So, you may need to wait for completion of body consumption by the ReadableStream due to backpressure, or it's ok that a UA fulfills the trailer promise without waiting for completion of consumption if possible.

not distinct

https://tools.ietf.org/html/rfc7230#section-4.1.2 says

   When a chunked message containing a non-empty trailer is received,
   the recipient MAY process the fields (aside from those forbidden
   above) as if they were appended to the message's header section.

Not fully sure, but I interpret this as it doesn't forbid distinction between headers and trailers at the application level.

[annevk]

@mnot no, we don't want combine semantics at all. We just want a distinct Headers object. A response would have "headers" and it would have "trailer headers". The user agent would only ever look at "headers". Script can look at "trailer headers" too.

[annevk]

So trailer headers are atomic just like normal headers are atomic? How are streams affected?

Also, I thought that semantically trailer headers are not distinct from normal headers, though it's not entirely clear to me how that's supposed to work in a streaming situation.

[annevk]

Given that section 4.1.2 it seems we also need to define what happens when a server does generate those forbidden trailer headers. Will we just pass them through or discard them. And if we discard them, what is our whitelist/blacklist?

I think the UA should fulfill the trailer header promise once the network layer has consumed all trailer header bytes and parsed them successfully. I don't think it needs to depend on content consuming a stream, but I might be missing a subtlety.

[louiscryan]

You should also consider how you want to allow client generated requests to produce trailers. As chunked-encoding must be used in this case and this would have to be specified before the body is produced for HTTP/1.1.

https://tools.ietf.org/html/rfc7230#section-4.1

Separately the client needs to indicate whether it is willing to accept trailers in the response using "TE: trailers", specifying this via the normal request headers mechanism seems sufficient.

https://tools.ietf.org/html/rfc7230#section-4.1.2

Given that trailers are strictly limited to occurring after the last transfer chunk and that the content of the trailers may vary based on the application behavior right up until writing the last chunked byte it seems the API should not require the trailer values prior to receiving the last byte on the stream.

A decision also needs to be made about how to interpret the "Trailer" header itself, though given that it's a SHOULD in the spec I suspect it should just be propagated with the initial header values rather than using it to implement some strict presence checks for the subsequent trailers

https://tools.ietf.org/html/rfc7230#section-4.4

[domenic]

I think streaming should not be too affected. It makes sense to fulfill the trailers promise as soon as they are available---which might be never if the stream consumer exerts backpressure and stops the flow, or might be before the stream consumer fully reads all chunks if the stream is doing internal buffering. And assuming they are used for similar metadata to headers I don't see a need to stream the trailers; we can use the same model as headers and get them all at once.

[annevk]

@louiscryan I think what @tyoshino suggested for requests makes sense. You supply a promise at the point you initiate the request. That indicates ahead of time you might supply trailers.

[louiscryan]

Agreed. Promises seem like a good way to model the intent even if the value
is never used as a result of error.

[annevk]

Another thing we need to define is the relationship between a response having a trailer and the load event for various (non-fetch(), fetch() doesn't do events) features. Does a trailer (that potentially stalls) delay the load event or not? In other words, what's the relationship with process response end-of-file. I guess it would happen after and we'd introduce process response trailer?

[tyoshino]

Separately the client needs to indicate whether it is willing to accept trailers in the response using "TE: trailers", specifying this via the normal request headers mechanism seems sufficient.

It's forbidden to set the TE header via the request headers construction API. Need a method to instruct the Fetch to add it. Maybe a boolean in the RequestInit?

[annevk]

@tyoshino ah yes, for the response we would need that. Perhaps requiring either responseTrailer : "preserve" or supplying a trailer argument with a promise as per your strawman.

[louiscryan]

Not sure I follow your reasoning. Trailers are part of the spec and the historically poor support in server implementations is in no small part because there was no browser API which this proposal would address. Trailers do have support in a non-trivial number of the more popular server libraries and proxies. HTTP2 does a reasonable job addressing the encoding issues that have also dogged trailers.

Is the functional utility of trailers if they were well supported in question? E.g. MACs, trailing status & debug info.

[martinthomson]

If the use cases for trailers were actually compelling, then we would have seen client implementations by now. And it's not just browsers that matter here, you have to convince developers of major client libraries (I'm thinking Windows, iOS and Android here) that it's worthwhile too.

"Build it and they will come" is a posture that ignores the fact that this feature has gone 15 years without any real traction.

[louiscryan]

Last time I looked library support was actually pretty decent

.Net https://msdn.microsoft.com/en-us/library/system.net.http.headers.httprequestheaders.trailer(v=vs.118).aspx
Plain Java (URLConnection, HttpClient, Netty, Jetty & most servlet implementations)
Android Java (OkHttp & URLConnection(?))
libcurl

Ill let others weigh in on iOS

[igrigorik]

+1 to Mark's comments. I hope that HTTP/2 is an opportunity for us to rally around enabling proper trailer support in clients, not the other way around. The "lack of real traction in past X years" is a circular argument: use cases are blocked on support, support wants to see "real-world" use cases.

FWIW, Server Timing (http://w3c.github.io/server-timing) would benefit a great deal from trailers.

[sleevi]

While Martin phrased it as "actually compelling", I would like to put out the other argument - experience is that there are profound security implications regarding how you operate on data and if and whether transformative operations are allowed to be sent in trailers that cause the content to be reinterpreted.

For example, a big concern I spelled out in tc39/proposal-cancelable-promises#4 was precisely regarding header truncation, with trailers offer (both implicitly - through poor implementations - and explicitly - via cancelable promises). I have little faith that trailers have received proper security analysis, especially given the past two decades of security research, with my gut (and my bias) being that they represent yet another attack vector for folks like @sirdarckcat in browsers being too clever for their own good.

If there were demonstrably compelling reasons, then I'd hope it was first explored outside the context of any programatic API, so that proper security review and experience with could be accomplished. But I'm really not keen to even see this implemented in the browser - the gain vs risk is too disproportionally balanced in the latter.

[igrigorik]

@sleevi fair points, but I'm reading this as "we don't know what we don't know so we shouldn't do it", which seems like the wrong place to start. We do know that there are concrete use cases that developers would benefit from (e.g. trailer ETags, timing metrics, and so on) and I think we should pursue this with appropriate security reviews and guidance.

[sleevi]

@igrigorik You're reading it wrong.

I'm explicitly opposed to exposing it in fetch() until implementations have worked through the security implications. I'm implicitly opposed to implementing at all, given the complexities in doing it correctly (on both ends) for the value it returns (thus agreeing with Martin on questioning the use cases as compelling). But you should read the explicit greater than the implicit, and you should replace seeing it as FUD (although there is each of those) with "This has been repeatedly botched with stream aborts, and this adds a whole new attack surface, so there should be demonstrable value"

[igrigorik]

@sleevi fair enough, I agree that we should (carefully) work through the security implications. I just want to make sure that we don't end up burying this prematurely (once again :)).

[louiscryan]

Am I right in assuming that the objections are limited to the additional risk interpretation of truncated trailers present beyond the risks already present in truncating HTTP/1.1 chunked transfers / HTTP2 DATA frame sequences?

Similarly we're only concerned with exposure/interpretation of these headers in the API distinct from whether receipt of trailers from the wire impacts whether a response is considered valid or not. AFAIK most XHR implementations simply drop received trailers on the floor but do not consider their presence an error at the API level.

I don't claim to know the intricate details of how browsers interpret headers and which headers would present a greater risk when truncated after some payload has already been received. You mention 'transformative operations' earlier, could you clarify which headers you consider to present a risk in that class?

[sleevi]

@louiscryan No, that's not correct - that is, the objections are not limited to that. It's about a scope of authority separation that trailers introduce.

Consider, for example, RFC 6797 or RFC 7469. Both of these explicitly restrict processing to the first instance of the header field (e.g. http://tools.ietf.org/html/rfc6797#section-8.1 http://tools.ietf.org/html/rfc7469#section-2.3.1 ). This wasn't accidental, but predicated by a concern about the separation between the capabilities being expressed by someone who controls a given resource and those who control the domain/server. In many (most) systems, the headers are handled by a 'trusted' system (such as the web server), which handles access controls about what can be set where and by whom, and then shunts processing of the body off to other, less-trusted systems (think CGI scripts, although I'm sure you can insert your favourite framework here). The trusted process handles the preamble generation, and then everything else is handed to the untrusted process and no further parsing/inspection is needed by the server.

To put this more concretely, consider a server that wished to delegate the production of content bodies to some 'untrusted' process, which may include the production of chunked bodies. In order to safely sanitize headers (especially security-relevant headers), the server implementation would need to inspect/process all chunks and filter them along the way, versus the current behaviour of filtering them up at the forefront.

This is just one example where the state of non-implementation in browsers is a security benefit to server operators. While it may be perfectly fine when all elements of the data production are trusted (such as the cases @mnot raises), I don't believe that the threat models are the same (nor do I think Mark would suggest they are), thus we can't reach a conclusion that what is good for the goose is good for the gander.

This sort of system, as a concept, is a common source of attacks and vulnerabilities. For example, consider certificates that contain multiple common names. Some CAs would only validate the first instance as a domain name, and let the remaining instances be requestor-controlled. However, some applications would check the first CN, others would check the last CN, and there would be confusion such that those who looked at the last CN would be seeing attacker controlled data. Or, to use another certificate-based example, Moxie Marlinspike's null termination attack (which related to ambiguity whether systems checked the name from the beginning up to the null, or checked the name from the end).

Even within HTTP, you've had situations like Request/Response smuggling vectors leading to confusion.

I hope you can see that there is a whole host of systemic errors that emerge when there are multiple ways to represent something, when implementations may support one, the other, or both, and when there is additional filtering overhead beyond "out of sight, out of mind". This is why we (intentionally) have not supported trailing headers and are loathe to do so, and why it's such an exceptionally high bar outside of niche use cases that may not have the same security concerns.

[davidben]

I agree we should just give up on trailers in the browser context at this point.

There's first the huge mass of interop and security pitfalls. Browsers process existing headers everywhere. When I think about where we process them and what the headers are, they span a spectrum from:

• "We could accept this in a trailer, but what's the point?" (Alt-Svc, HSTS, HPKP, Connection, Refresh)
• "This might work, but there would be very surprising consequences" (Set-Cookie, Link)
• "I can understand would this would mean, but the complexity to the system is not worth it" (Location, all cache-related headers, auth headers)
• "This cannot possibly work, it MUST be in the header" (CSP, X-Content-Type-Options, really just about every header)

Not to mention the unbounded use of headers in web content. We can't make XHRs suddenly see trailers mixed into headers without warning.

Given that all existing code and headers have been designed without trailers in mind, I think the only answer is that we MUST break the correspondence between header and trailer semantics. Trailers and headers must be considered completely unrelated animals. Unless a field is explicitly specified as a trailer, it will be treated as any unknown field in a trailer. [Edit: Fixed some confusing wording here.] Anything else welcomes unexpected security problems, bizarre behaviors, and surprise interop failures when two implementations disagree on which headers would be too hard to process streaming.

Even after accepting that, there are significant costs. The current core data model for a network request is very simple: you have an atomic set of request headers, sometimes a body stream, and the net stack gives you back an atomic set of response headers followed by a body stream. There's side fluff around redirects and auth hooks and MIME sniffing and such, but that's the core abstraction. It's very clean. We can build a clear division of responsibility between the atomic response headers and streaming body. This is how navigation works, this is how some servers work (as Ryan noted), etc.

Changing that core data model is not a small change. This affects not just fetch, but everything in a browser's network stack from the low-level HTTP implementation to the disk cache to any hooks (extensions, etc.) to web content to all the layers in between.

I think the use cases need to be exceedingly compelling to entertain this.

[louiscryan]

@sleevi I take your point about the separation of trust between header production & 'everything afterwards' in some systems but I don't think anyone should be relying on browsers as the last-line of defense against these issues. Caching / De-chunking proxies are free to promote trailers into headers & HTTP/1.1 keep-alive requires servers to observe a chunked response for termination.

I note that a lot of the discussion here is focused on how the browser should interpret and enforce policy around trailers when things like (Alt-Svc, Set-Cookie, HPKP are present in trailers). With the possible exception of ETag / Content-MD5 (and @mnot my have some others) my assumption was that the browser would continue to ignore headers from an internal processing perspective just as they do today. That is to say that trailers have no semantic meaning to the browser and they are simply made available for programmatic use by developers in the fetch API.

@davidben - I don't think anyone is suggesting mixing trailers into headers in XHR. That's a de facto standard and changing it's contract would be a very bad idea. The proposal is to allow for them in fetch possibly as en explicitly separate feature from headers as it's an entirely new interface and so would not disrupt existing usages.

[davidben]

I certainly agree that a precondition here is that all existing headers (including ETag and friends; allowing that is considerable complexity to a disk cache implementation) must be ignored in trailers. Though, even that might be troublesome. The client advertisement is simply "trailers", not a list of headers that may be put there. I don't know how enthusiastic existing servers will get if you start advertising support. Will ETags suddenly move to the trailer? What will happen if some browsers do support ETags in the trailer and some don't?

Even if that turns out fine (the install-base of servers probably does not include that many trailer users?), this is still a massive change to the core abstraction of the whole browser. The benefits have to outweigh the cost. If the use case is purely programmatic use by developers, it seems this is not the right time to propose the feature. The core capability (a stream of bytes) is already being provided by Fetch. In the spirit of the extensible web manifesto, JS code is free to parse out whatever additional structure it wishes. While we don't let you extend the HTTP parser, the difference between that and parsing out the body is minimal.

If it turns out everyone is effectively reimplementing HTTP trailers (unlikely), then it makes sense to move a construct of the sort into browsers. The reason one might wish to short-circuit this process is if some processing in the browser needs to interpret this. But we seem to both agree that the opposite is desirable.

(Nit: It's not accurate to say XHR lacking trailers is a de facto standard. This is de jure at this point. The specification is Fetch and friends. They say, on the web, HTTP trailers do not exist. This ticket exists specifically to propose changing that.)

[louiscryan]

I wouldn't say that the parsing separation is minimal overhead in the case of handling stream termination. JS code will rely on the HTTP parser for non-streaming status handling and stream initialization handling but will have to switch to payload parsing for stream termination status.

A payload and therefore it's mime type will have to describe how terminal statuses are encoded and make them reliable for the range of possible termination causes. Consider an API like the Twitter streaming API which returns a homogeneous sequence of types in a stream

https://dev.twitter.com/streaming/reference/get/statuses/sample

If they wanted to convey a termination status after a period of time like 'Credential Expired' or 'Budget Spent' to a browser they are forced to change their schema. Today they just hard drop connections https://dev.twitter.com/streaming/overview/connecting which is not exactly pretty.

Trailers solve this problem in a way that is also not subject to streamed message truncation causing the developers payload parser to get into an unrecoverable state which prevents it from parsing the special 'error' entry in the stream.

It also wouldn't require new MIME types for commonly streamable formats like audio & video that want better termination handling though these are so valuable on the web that side-polling is used to bridge the gap when 'Rental Period Expired' kicks in :)

On the subject of compelling use-cases what kind of quantification would be persuasive?

[davidben]

I'm not sure I follow. Trailers don't have any magic as far as termination is concerned. If the connection to the server is shut off, we're not going to get the trailers either. HTTP/2, for all the bells and whistles, is still sent over a byte stream.

Changing the schema for a termination signal sounds about right? You're sending a sequence of objects terminated by another object. Trailers just make this more confusing. Now, as a client, I have to care about what it means if the peer sends half a status and then stops with a trailer.

In the case of extra formats, this proposal isn't going to help the video tag anyway, no? It's only if you do a low-level fetch() in JS. That data needs to be plumbed your video elements some other way (Media Source Extensions?). I dunno, if you're so set on trailers, you could just completely emulate them with a single outer MIME type which parses to a wrapper stream of inner MIME type + data + trailer. The outer MIME type really doesn't matter much because it's only processed by your JS code anyway.

It seems, even if fetch() gets this feature, you need this wrapper type anyway. Say your Twitter example wanted to use trailers. They still need a story for older browsers. (Given how incredibly invasive this change would be, I'm dubious every browser will be convinced, certainly not without demonstrated need.)

If there's no fundamental new capability, these kinds of polyfills are really the right way to get this kind of APIs into a browser. They are already necessary for deployment and can evolve much faster into the right solution for some use case, rather than the first one we thought of at the time. Successful JS experiments demonstrate need and use-cases. Those beget candidates for browser APIs. (Only candidates, not guarantees; again, changing the core data model like this is not a small change. This has one of the highest bars a network feature could have.)

Remember, HTTP trailers do not exist as far as browsers are concerned. You have to imagine this as if someone's proposing adding a brand new unheard of HTTP semantic to browsers. That they're already spec'd is, if anything, a drawback. That means the latent threat of compatibility problems is worse because the install-base may already try to speak it.

[louiscryan]

The point was that its preferable to not have to change the schema and unambiguously use trailers to convey terminal status. This allows for graceful upgrade of existing APIs that currently can't represent a meaningful terminal status without breaking backward compatibility. The story for older browsers using the Twitter API is unchanged.

Using a MIME multipart of application/http can and does work, I've done such things before ...
https://developers.google.com/drive/web/batch#example
... but it performs terribly and negates many of the benefits of HTTP2/SPDY not to mention the lack of routing or header injection capability in reverse proxies.

[mnot]

@sleevi - I'm reading in a hurry in an airport lounge, but with the caveat, it feels like you're arguing against something that is not this proposal. AIUI this bug is not proposing that trailers be folded into existing headers willy-nilly (which indeed would be insane); rather, it's making them available to applications that specify the use of trailers, and (presumably) understand the various security risks.

Now, one might argue that trailers are Just Too Dangerous to expose to any application, even with full knowledge. However, I'd find this a might curious place to draw that line, given where we're at.

To give an example, I received an e-mail from one of your Googly brethren just yesterday asking about how to put a digest into trailers for integrity checking purposes. I'd design that by defining a header that communicates the algorithm, and a trailer that carries the actual digest. Are there security and interoperability issues in that use case? Certainly, but it's not being designed without trailers in mind.

[annevk]

Indeed, all this is proposing is that we expose new a header name/value map on responses which has no semantic implication for responses whatsoever. That is, this is only useful for the layer which invokes fetch() and wants to get some post-body information from the server. It won't impact anything else. Everything else will keep ignoring trailer as per the current standard.

[igrigorik]

Remember, HTTP trailers do not exist as far as browsers are concerned. You have to imagine this as if someone's proposing adding a brand new unheard of HTTP semantic to browsers.

@davidben nitpicking on language, but they do exist in the sense that today we process but ignore them. At a minimum, we could/should make them visible, as that alone will enable plenty of interesting use cases.

[davidben]

Fascinating. I had not realized servers were allowed to send those without receiving TE: trailers header (which browsers do not send), but you seem to be right. That seems a rather bad protocol bug. I wonder if we're actually stuck with it... oh well.

Anyway, making them visible isn't really a task to do "at a minimum". That's where the cost comes from. To expose them, everything in the very very long pipeline needs to change. (Consider the disk cache. We don't store trailers there. We cannot expose trailers and not store them in the disk cache or the disk cache's abstraction breaks. So that requires redesigning the disk cache to track them. Also some kind of migration story for old cache entries that lack the trailers. This is just one example in the pipeline.)

[mnot]

They're allowed to send them, but with the limitation that such trailers can (and do) get dropped on the floor, either by an intermediary, or by a client that doesn't do anything with them.

[igrigorik]

@davidben fair points. I wonder if this applies to other implementations. /cc @mcmanus, @toddreifsteck

I'm wondering if it would help to nail down some concrete use cases and then examine how/if we can support them and their associated costs and risks. For example, in the context of using trailers for communicating timing metrics, cache persistence is a nice-to-have and I think we could live without it.

[ricea]

As @louiscryan mentioned, proxies are free to promote trailers to headers. This is described in http://tools.ietf.org/html/rfc7230#section-4.3

As a result, HTTP trailers cannot be re-purposed as a fetch-specific metadata mechanism, since proxies may turn the trailers into ordinary headers. In my experience, there exists no feature so terrible that some proxy somewhere won't implement it.

The other consequence is that it is not safe to add trailers to a message on the assumption that the browser will ignore them; an intermediary may promote them to headers at which point the browser will process them normally.

My interpretation of these facts is that trailers can never be safely used on the public web for any purpose, and should be actively deprecated.

[mnot]

The question is not whether you can "add trailers to a message on the assumption that the browser will ignore them"; it's whether you can design a feature to use them safely. Features can (and should) be designed so that they can safely handle when something expected to appear in a trailer instead occurs in the headers.

Furthermore, preventing access to trailers in browsers won't change the behaviour of proxies at all.

[louiscryan]

Proxies promoting trailers into headers is a pre-existing issue and the proposal here would do nothing to change that. I raised the issue in response to sleevi@ noting that browsers interpreting trailers may pose a new security risk simply to note that the risk already existed in another form and nothing being discussed here is addressing that issue. Relatedly no one has argued for browsers to start interpreting trailers, which would represent a new risk.

Given that terrible proxies are already a feature of the internet and are already causing problems how does the proposal here make that situation any worse? Is the argument that by browsers exposing an API for trailers it will encourage their use and therefore increase the aggregate risk posed by poorly implemented, as opposed to actively malicious, plaintext proxies?

[annevk]

We could restrict this feature to secure contexts. Kills the public proxies. (Not that I think that concern matters much for what is being proposed here, but...)

[mnot]

That's a bit arbitrary (the security aspect is in how the trailers are used), but that said I'm not against such a limitation.

[louiscryan]

I'm not particularly against it either but I do wonder about the practicalities for application developers prior to deployment. I guess the definition of 'secure context' is key here.

[igrigorik]

I'm not sure the secure context restriction is meaningful. First off, nothing stops the other end of your secure tunnel from promoting trailers into headers, and if that happens then we already process them if they're meaningful -- e.g. my origin streams a response and appends an ETag trailer, and my CDN buffers the response and promotes the ETag to a header; the browser will process and use the ETag value for future revalidation.

[martinthomson]

@igrigorik, it's absolutely not meaningful in the sense that it would provide a security benefit, but at least on the Mozilla end, we're trying to avoid shipping new functionality without secure contexts.

[igrigorik]

@martinthomson fair enough, that a policy decision that's out of scope for this discussion. With that, I think we're back to @annevk's earlier comment:

Indeed, all this is proposing is that we expose new a header name/value map on responses which has no semantic implication for responses whatsoever. That is, this is only useful for the layer which invokes fetch() and wants to get some post-body information from the server. It won't impact anything else. Everything else will keep ignoring trailer as per the current standard.

As a further constraint, I think we identified that this name/value map may not be persisted by the HTTP cache. Anything else?

[sleevi]

@igrigorik I do want to make sure you're not ignoring @davidben & I's concerns about the fundamental complexity this would bring to a browser implementation to even support. While I think it's fine to discuss "What might the API for fetch() look like to expose this", it still fundamentally requires that the UA be able to understand and expose this, and thus the networking stack support it, and I think the complexity/use case tradeoffs are far, far from being met. But I suppose that's a broader question of whether UAs should be expected to support trailers, period, rather than the exact shape of the API.

Just making sure we're not in a situation where we're haggling over the colour of the bikeshed, when we haven't even agreed upon the need for a bikeshed.

[mcmanus]

I have deep concerns here too.

It sounds like more than "http trailers" it seems there is a desire to have some kind of meta-data that might not come first. Could a particular definition of a h2 push stream satisfy those semantics more safely?

[annevk]

@mcmanus yes, some way to deliver metadata about a resource after the resource has been generated and transmitted. Metadata that has no processing implications for the resource.

I would be fine with not doing it through trailers if none of the network stack folks are comfortable with that.

[igrigorik]

@sleevi fwiw, I don't think I am ignoring the raised concerns... As I noted before, existing implementations already "handle" trailers: we don't error out when we see them, we simply drop them on the floor. As such, if we add further carveouts that such metadata doesn't make into the HTTP cache (i.e. existing behavior), or affects the actual processing in any other way, what other implementation concerns do we have?

As @annevk mentioned, the only thing we're asking for here is that the trailers are parsed and exposed in a new map on the response object.

@mcmanus how would that work? Would I push a same-URL resource as the parent (err?) with headers at the end?

[davidben]

We can't have a cache carveout for trailers if we expose them. That we drop them on the floor is precisely the reason it's okay that we don't put them in the cache. As soon as that changes, this no longer holds.

Imagine a resource which sends max-age=99999999 and sends a trailer. So you fetch() it, you read out the trailers. Random JS code does whatever it wants with said trailer. Hooray.

Now we fetch() it again. The cache entry is used without revalidation. We didn't save the trailers, so they don't get sent up this time. And now the JS code is confused because the trailers suddenly disappeared.

[igrigorik]

@davidben yep. Perhaps I'm in the minority on this one, but as long as that behavior (i.e. trailers are not cached) is clearly expressed in the contract, I don't think there is anything wrong with that.

[mcmanus]

@mcmanus how would that work? Would I push a same-URL resource as the parent (err?) with > headers at the end?

one way it could work would be to push "/.well-known/trailers" with a json content-type - (it is scoped to the stream being fetched by virtue of the push promise frame being the same stream id as the object being fetch()d). The PP has to come early but the actual json does not and could be empty or the promise stream even rst if you had nothing to say at the end.
.

[igrigorik]

@mcmanus not sure I follow.. AFAIK, per spec the server can't reply with same ID? Say I have...

• Client request: {id: 5, path: /thing}
• Server issues PUSH_PROMISE: {id: 6, path: /.well-known/trailers, body: some-json-thing}

We can't use the same path for all trailers since that'd clash with existing caching logic and streams would trample over each other. How do we tie these streams together? Assuming that's resolved, looking up trailers would require a separate cache lookup? Or, do you have something different in mind?

[mcmanus]

@igrigorik

Client request: {id: 5, path: /thing}
Server issues PUSH_PROMISE: {id: 6, path: /.well-known/trailers, body: some-json-thing}

not quite.. server issues
PUSH_PROMISE: {id: 5, promise id 6, path: /.well-known/trailers, body: some-json-thing}

pushes promises are part of an open(ish) peer initiated stream

[igrigorik]

Would the cache then have to take stream ID into account when creating the cache-key for ./well-known/...? Also, how would I access it as a developer -- would the UA automatically look it up and make it available as a property on the Response object?

Rereading the spec, there is also this gotcha:

It is not possible to push a response to a request that includes a request body. - 8.2.1

... which means this mechanism would be restricted to "safe" methods only, and if we do allow response bodies on GET then that breaks too =/

[louiscryan]

That approach seems pretty problematic for a number of reasons

The Fetch API has no explicit API to correlate streams based on id so correlation in the application layer has to be done on resource URL. Given that PUSH_PROMISE stipulates that the response of the pushed stream is cacheable (8.2.1) if you have two active streaming POST responses from the same URL you would have a hard time determining which push-promise entry in the cache was associated with which active request and would likely be forced to implement some higher-level scheme again.

Not only does this require HTTP2 end-to-end but it also requires full PUSH_PROMISE support in all intermediaries. This is very high bar for deployment today.

[mcmanus]

rereading the spec, there is also this gotcha:
It is not possible to push a response to a request that includes a request body. - 8.2.1

the request (body) in question here is not the associated stream that started the fetch (stream 5 in our example) - it is talking about the request for the pushed stream (6 in the example). Every transaction has a request and response, for a pushed stream the request headers are in the push promise (in this case something like GET /.well-known... always GET even if the fetch was a POST) and the response headers are in a normal headers frame (bearing stream id 6). So the RFC requirement says the push promise request headers cannot have a request body (indeed it has no way to express one) - but the associated stream (the fetch) certainly can.

Would the cache then have to take stream ID into account when creating the cache-key for ./well->known/...? Also, how would I access it as a developer -- would the UA automatically look it up and make it available as a property on the Response object?

From an HTTP level, I would define the pushed metadata response as uncachable and then proivde a definition for caching fetch() that is bigger than just mapping to the transaction that started the fetch. 8.2 specifically says you can have an uncachable pushed response with a lifetime that matches the associated stream "Pushed responses are considered successfully validated on the origin server (e.g., if the "no-cache" cache response directive is present ([RFC7234], Section 5.2.2)) while the stream identified by the promised stream ID is still open." - so you can use it to satisfy the metadata aims of a particular fetch() but you don't need to worry about it ending up polluting the namespace of the general cache.

[mcmanus]

The Fetch API has no explicit API to correlate streams based on id so correlation in the application layer has to be done on resource URL.

I presume the fetch API would just define a property on the response object (tailingMetadata or whatever). The implementation would do the correlation based on the stream IDs. The URL would just be to match a name provided by the fetch specification so that you know it is providing trailingMetadata and not some other piece of information.

Given that PUSH_PROMISE stipulates that the response of the pushed stream is cacheable (8.2.1)

it doesn't say that afaict. what are you referring to? 8.2 talks specifically about how non cachable responses can be given to the application (the application in this case is the fetch() implementation) but not placed in a generic cache. I think that's fine for this use case.

if you have two active streaming POST responses from the same URL you would have a hard time determining which push-promise entry in the cache was associated with which active request and would likely be forced to implement some higher-level scheme again.

The appropriate request is obvious to the fetch() implementation (not the user of the fetch api) because it has visibility into the stream ids

[igrigorik]

Not only does this require HTTP2 end-to-end but it also requires full PUSH_PROMISE support in all intermediaries. This is very high bar for deployment today.

I agree. Also, intermediaries are allowed to drop pushes, which means that your trailer meta-data may disappear without any warning, which is problematic.

Re, caching: we'd have to special case caching of 'well-known' URL too, since per spec we can define different caching strategies for parent and the pushed resource; but it would be really odd for the well-known URL to outlive the parent, or for the well-known URL to expire before the parent.

@mcmanus stepping back, it's still not clear to me how this route is better (less risky, less complex?) than using the built-in trailer mechanism? The code would have to handle trailers regardless, albeit today we just drop that data on the floor. What are your concerns with parsing it and making it available to the application?

[mcmanus]

@igrigorik my suggestion was based on the argument that trailers are not appropriate to use as they are too easily conflated with headers along the way. That seems to be more or less what rsleevi and davidben are arguing.

Based on that constraint, I argued that what you really require is not trailers but rather non header meta data and I observe that H2 has a mechanism for sending N blobs linked to a single client request (PUSH). Obviously, not everybody thinks about the mechanism in that way but I'm actually pretty excited about it. Its on my TODO list to propose a way to surface this generically in web space (there is a priv'd interface for Firefox bits to get it already). This is an obvious way to scope pushed notifications - for example.

The proposal here (which I'm not invested in, I've just enjoyed explaining some corners of the spec.. its like a ietf whatwg mashup :)) - uses a special case.

Its not hard to imagine this as a name/value map of N pushed resources rather than just "trailers".

fwiw I wouldn't expect you would cache anything under the well-known URL cache key - you would add it to the fetch()d response on the side as some kind of metadata.. the same way you would presumably end up storing h1 trailers (because mixing them with headers is dangerous).

as for the argument that it requires new technology: guilty.

[igrigorik]

@mcmanus thanks, that fills in the gaps :)

my suggestion was based on the argument that trailers are not appropriate to use as they are too easily conflated with headers along the way.

I guess that's the crux of the issue, but I think that still brings us back to #34 (comment):

Proxies promoting trailers into headers is a pre-existing issue and the proposal here would do nothing to change that. I raised the issue in response to sleevi@ noting that browsers interpreting trailers may pose a new security risk simply to note that the risk already existed in another form and nothing being discussed here is addressing that issue. Relatedly no one has argued for browsers to start interpreting trailers, which would represent a new risk.

[mnot]

The most conservative thing that could be done would be to target a handful of headers that we determine are safe to combine (e.g., timing information, ETag) and specify them to be processed by browsers. That would give some experience with this and meet some use cases; the general mechanism could be added later.

[igrigorik]

A response would have "headers" and it would have "trailer headers". The user agent would only ever look at "headers". Script can look at "trailer headers" too.

+1

[louiscryan]

+1 largely because were talking about the fetch API here.

Speaking as someone who serves a LOT of ETags based on buffering content I would like to see UA support for that in trailers too, I'm just not sure this is the right place to discuss that (is there a place to discuss that??)

[annevk]

@louiscryan Fetch is the interface between user agents and HTTP (and other things), it's not just an API. Having said that, any standards forum is typically a bad place to convince user agents to implement a feature they don't want to implement.

[annevk]

Clarification on 1, for now we'll add it to responses. Since adding it on requests would also require chunked encoding which browsers don't do at the moment we'll leave that once request streams are sorted.

[annevk]

@yutakahirano do you have suggestions for how to modify step 14 of https://fetch.spec.whatwg.org/#concept-http-network-fetch to account for bytes that are part of the response body and bytes that end up forming the trailer? I guess we should just tweak the language a bit and maybe add a clarifying note that the framing in the protocol takes care of the distinction...

[yutakahirano]

Currently the description expects that Transfer-Encoding is processed at a lower layer. Can we assume that such a lower layer decouples the message body part from the trailer?

14.2: Whenever one or more bytes for message body are transmitted, ...
14.3: If at any point the bytes transmission for message body is done normally and stream is readable, ...
14.4: If at any point the response trailer is available, set it to response.trailer. // some notification is needed
14.5: (original 14.4)

[annevk]

I used payload body in #344, but I think you're correct I should update some instances to say message body instead. I don't explicitly set response's trailer since we don't explicitly set the other fields of a response either (except body...).

[mnot]

WRT TE: trailers, see also: httpwg/http-core#18