Better recovery when POST requests fail due to connection pool problems about okhttp

[clelland]

My understanding of this situation is that it is impossible, at the Java layer, to detect that the server has closed its side of the TCP socket, until we try to read data from it. I think that right now, the entire request is being sent, regardless of its length, and the EOFException is not being thrown until we try to read the status code from the response.

At the TCP layer, the client socket is sending several packets out to the server, which will sent RST packets back as soon as it receives them. Once the client gets the first RST, then I believe that it stops sending data out the network (it may be just black-holing what it receives from the application layer) but by then, quite a bit of data could have gone out, depending on the link latency and the window size.

It would be great if we could do a non-blocking read() on the socket before attempting to send, but I don't think Java sockets will let us do that; without switching the whole HTTP layer to nio; we may just have to send the whole request.

[swankjesse]

I'm changing the code to silently recover from failed connections. This class should make it possible to recover if the failure occurs while we're streaming data to the server:
#138

[swankjesse]

#153

[swankjesse]

@clelland how would doing a non-blocking read help?

[clelland]

I'm pretty sure that doing a non-blocking read() call on a socket which has been closed will return 0 (to indicate EOF). At that point, we would know to close our end and reopen the connection.

The problem is that reading a java.net.socket is always going to wait for data; it has no way to know that the other end is closed until you write to it (and it tells you with an exception)