Chunked may not be final in transfer encoding list about http-parser

[sam-github]

I don't understand what you think the issue is.

chunked, gzip may look meaningless but it is possible and its behaviour is well described.

Yes, its behaviour is described:

If a Transfer-Encoding header field is present in a request and the chunked transfer coding is not the final encoding, the message body length cannot be determined reliably; the server MUST respond with the 400 (Bad Request) status code and then close the connection.

[andrew-aladev]

@sam-github, this exception is assigned for the message body length cannot be determined reliably + chunked transfer coding is not the final encoding. It means that you can't receive HTTP header with Transfer-Encoding: chunked, gzip and without Content-Length.

But if Content-Length exists than Transfer-Encoding: chunked, gzip is valid header.

[indutny]

From the very same 3.3.3:

   If a message is received with both a Transfer-Encoding and a
   Content-Length header field, the Transfer-Encoding overrides the
   Content-Length.  Such a message might indicate an attempt to
   perform request smuggling (Section 9.5) or response splitting
   (Section 9.4) and **ought to be handled as an error**.

In other words, Transfer-Encoding cannot be present together with Content-Length.

---

It must be said that the presence of both headers leads to quite practical request smuggling attacks as was demonstrated by security researchers.

[indutny]

chunked encoder/decoder should be shipped outside http parser as separate library. It should be used by end-user together with gzip, brotli, zstd, compress encoders/decoders.

This simply can't work as the presence of chunked TE alters protocol in a way that cannot be offloaded to a 3rd party module.

[andrew-aladev]

Hello @indutny, I can't agree with you, please read carefully.

1. If a Transfer-Encoding header field is present in a response and the chunked transfer coding is not the final encoding, the message body length is determined by reading the connection until it is closed by the server.

2. If a Transfer-Encoding header field is present in a request and the chunked transfer coding is not the final encoding, the message body length cannot be determined reliably; the server MUST respond with the 400 (Bad Request) status code and then close the connection.

3. If a message is received with both a Transfer-Encoding and a Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request smuggling.

3 exception is related to request only, response just reads as much as possible data from server and has no possible security exceptions.

Please read RFC 7230 - 3.3.1 Transfer-Encoding for clarification too.

1. If any transfer coding other than chunked is applied to a request payload body, the sender MUST apply chunked as the final transfer coding to ensure that the message is properly framed.

2. If any transfer coding other than chunked is applied to a response payload body, the sender MUST either apply chunked as the final transfer coding or terminate the message by closing the connection.

This simply can't work as the presence of chunked TE alters protocol in a way that cannot be offloaded to a 3rd party module.

I see no collisions. Transfer-Encoding: chunked, gzip means just gzip after chunked. User are reading data, than applies gzip and than chunked.

[andrew-aladev]

I was wrong about Content-Length: it can't be used to identify the length of body if Transfer-Encoding was used, this usage is forbidden. But responses without Content-Length can have Transfer-Encoding with chunked in any position with other encodings.

[indutny]

Well, FWIW it is allowed for responses:

http-parser/http_parser.c

Lines 1904 to 1913 in 5c5b3ac

	} else {
	/* RFC 7230 3.3.3 */
	/* If a Transfer-Encoding header field is present in a response and
	* the chunked transfer coding is not the final encoding, the
	* message body length is determined by reading the connection until
	* it is closed by the server.
	*/
	UPDATE_STATE(s_body_identity_eof);
	}

[andrew-aladev]

Parser doesn't depend on gzip, brotli, etc it can't decode chunked for user, because there is another encoding on the top of chunked. User have to handle chunked by himself anyway.