Requests with Large POST Body Immediately Goes into Delay Cancellation Right After the Server Received the Request about playframework

[jrudolph]

Thanks for looking into it, @mkurz. In general, we had HTTP pipelining indeed enabled for a long time but (as you mentioned) aside from benchmarks it is not recommended to be used in practice and there is little actual usage from any kind of common client libraries in default settings. One reason it is not recommended, is that you get head-of-line blocking and it is much more likely to overload a server with requests that might not even be served in the end (if the connection crashes for whatever reason).

So, in general pipelining support is there but it is not tested to the same degree that regular HTTP/1.1 support is and that's why disabled it for 10.2.

Regarding benchmarking, if you want to be fast in the pipelining benchmarks, you need to expose the pipelining setting and only turn it on for the benchmark. Pipelining can be beneficial in certain situations and imo it's fine to show those off in a benchmark but it's not thought to be turned on by default.

[mkurz]

@raphaelNguyen Thanks for your detailed report and the reproducer. I was able to reproduce locally as well. I can confirm this problem occurs with Play 2.9 and 3.0.

At first I was a bit confused, because I also tried Play 2.8.21 with akka-http upgraded to 10.2.10, the same version that Play 2.9.1 uses: With 2.8 it worked, but with 2.9 it failed (like you said)... I took me quite a while to figure out what the problem is, like what the difference in Play's akka http server backend is in 2.8 vs. 2.9...

Turns out the problem only occurs when HTTP pipelining (RFC) is enabled... which actually was the default in akka-http 10.1.x, but was disabled in 10.2.x (akka/akka-http#3321) (and therefore also Pekko v1).

So... despite akka-http disabling it in 10.2.x, Play did not:

• #11918

this was done mainly because benchmarks did suffer from disabling it and I thought we can just keep the setting enabled, since it seemed it does not hurt and akka-http had it enabled since years... I didn't know this would cause problems. It seems that meanwhile akka's internal code changed in regards of http pipelining so that problem like yours can surface....
Given that basically HTTP pipelining is more or less used for benchmarks, and not supported anymore in any major browser, nor curl (see here) I think we are safe to just disable it as well, given that akka-http did so and they say it's an obscure feature which isn't tested well (see the comments the above issue). Also Netty does not even support it.

Anyway, can you try to set:

PlayKeys.devSettings += "play.server.akka.pipelining-limit" -> "1" // Play 2.9
PlayKeys.devSettings += "play.server.pekko.pipelining-limit" -> "1" // Play 3.0

in your build.sbt and let me know if this fixes your problem? Those lines disable http pipelining.

You also have to put it into your conf/application.conf file, because PlayKeys.devSettings are only for dev mode:

# Play 2.9:
play.server.akka.pipelining-limit = 1
# Play 3.0:
play.server.pekko.pipelining-limit = 1

If you can confirm this fixes the problem, we will disable http pipelining with next bug fix release by default.

[jrudolph]

Btw. I'm not 100% sure that this actually is only caused by pipelining.

In my investigations, there was some indications that code in Play uses the head stream operator to ignore a part of the request entity. Since head will cancel the request entity stream, using head is not safe to do if you want to return a response (or keep using the connection). This has been under-specified in Akka HTTP (see e.g. akka/akka-http#3458). The rationale is that you cannot cancel an incoming request entity in HTTP/1.1 on the protocol level, so taking the cancellation signal seriously means we have to close connection immediately. I cannot remember exactly any more but I assume that for implementation reasons, returning a response might still work (probably because of the cancellation delay) but I would consider this accidental. This would also explain why pipelining is more certainly affected: eventually the connection will be closed after a request entity cancellation and if you already have more requests in-flight on that connection, all these will be affected.

The real fix in play should be to make sure that head, headOption and all other stream cancelling operators from request entity processing (BodyPartParser) do not cancel the request entity stream if you still want to return a response. You could replace them with another operator that just slurps in and ignores the rest of the stream e.g. a general purpose GraphStage that replaces a downstream cancellation signal with ignoring the rest of the stream and put it somewhere in front of the head in the request entity processing.

Arguably, a behavior like this could also be provided in Pekko HTTP directly under a setting (but currently isn't).

[mkurz]

@jrudolph Thanks. I might investigate when I find time (I did so right now quickly... but not into depth). Given the issue is resolved with disabling http pipelining I just leave things as they are now. However, if you want to come up with a pull request where we can discuss things more detailed, I am very happy if you do so.

[raphaelNguyen]

Thank you for taking the time to investigate this issue, @mkurz, @jrudolph. I can confirm that turning off pipelining looks to have solved the issue I was experiencing both in the reproducer and my own application. I will be turning it off manually until play release a new version with pipelining off by default. Apologies for the delay in getting back to you.

[raphaelNguyen]

Can I ask if there might be some progress on this? Thank you for your time.