erikd-ambiata / test-warp-wai

test-warp-wai

This is a repo containing code to reproduce a file handle leak in versions 0.2.6 and 0.2.7 of Haskell's hs-connection libraries.

This has been tested with both ghc 7.10.3 and ghc 8.0.1.

Reproduction

I'm assuming that this will be reproduced on Linux, but it should probably also work on OSX.

Clone this repo and the change into the freshly cloned source tree and do:

• cabal sandbox init
• cabal install --dependencies-only
• cabal configure

You will now need two more terminal windows (three total) and you will need to change into the above source directory in all of them.

In the first window run:

cabal build && dist/build/dummy-https-server/dummy-https-server

which bulds everything and runs a trivially simple HTTPS server on port 3000.

In the second window run:

dist/build/test-warp-wai/test-warp-wai

which has two main threads, the first of which uses http-conduit to query the HTTPS server started in the first window. The second thread (which deliberately starts some time later) runs a trvially simple HTTP server. When it starts it will print the message "Warp server running on port 2000."

When the "Warp server running on port 2000." message appears, int the third window, run the shell script:

./hammer-continuous.sh

which uses wget to query the HTTP server running on port 2000.

Results

With the above setup using version 0.2.6 or 0.2.7 of the conection library, everything seems fine but then after a minute or two the hammer-continuous.sh shell script will fail with:

Wget connection fail. Exiting.

and the test-warp-wai program terminate after printing:

accept: resource exhausted (Too many open files)

If instead the programs are all built with version 0.2.5 of the connection library, no problem will be reported even after an hour or running.

test-warp-wai's Issues

Data leakage between network streams with wai/warp/http-conduit

Hi @snoyberg,

I hope its ok to tag you like this. I'm hunting down a particularly weird bug which I currently suspect is in warp, wai or http-conduit. I've raised the issue here and tagged you because I can't figure out how to make further progress tracking it down. Although I currently think its warp, wai, http-conduit or maybe even in TLS it could also be an issue in the GHC runtime system.

The back story is that I first noticed it in one of our internal systems which has a Warp based HTTP server (via Airship) and writes data to AWS (HTTPS) using amazonka which sits on top http-conduit.

When I ran my server and ran a wget or curl request against it, I would occasionally get a packet like the following:

00000000  15 03 01 00 20 c7 40 42  0a 54 86 1f 8e cc 84 87  |.... [email protected]......|
00000010  b0 e3 0a 86 ec ba aa 37  3a 65 2e 57 3b fa 8d 09  |.......7:e.W;...|
00000020  58 f7 1e 36 cd 48 54 54  50 2f 31 2e 31 20 32 30  |X..6.HTTP/1.1 20|
00000030  30 20 4f 4b 0d 0a 54 72  61 6e 73 66 65 72 2d 45  |0 OK..Transfer-E|
00000040  6e 63 6f 64 69 6e 67 3a  20 63 68 75 6e 6b 65 64  |ncoding: chunked|
00000050  0d 0a 44 61 74 65 3a 20  54 75 65 2c 20 33 30 20  |..Date: Tue, 30 |
00000060  41 75 67 20 32 30 31 36  20 30 33 3a 33 39 3a 35  |Aug 2016 03:39:5|
00000070  32 20 47 4d 54 0d 0a 53  65 72 76 65 72 3a 20 57  |2 GMT..Server: W|
00000080  61 72 70 2f 33 2e 30 2e  31 33 2e 31 0d 0a 41 69  |arp/3.0.13.1..Ai|
00000090  72 73 68 69 70 2d 54 72  61 63 65 3a 20 62 31 33  |rship-Trace: b13|

As you can see, the response I should be getting starts at byte offset 37. The data before that is garbage, or so I thought. I spent a significant amount of time tracking it down before I figured out that if I disabled the component that uploads to AWS, the problem disappears.

I then noticed that garbage bytes that are prepended to the response is actually a TLS alert packet:

15             Alert protocol type
03 01          SSL version (TLS 1.0)
00 20          Message length (32 bytes)

Somehow, data from the http-conduit connection to AWS (over HTTPS) is leaking into the HTTP stream. I managed to confirm this using Wireshark. I can actually capture packets that contain one or more of these TLS alert packets (complete, according to the message length field) followed by my expected HTTP response.

I've also spent many hours working on a reproducable test case I can send you. Unfortunately re-producing what I reported above (the TLS alert packet prepended to a legitimate HTTP response) was difficult and took much experimentation and tweaking. The program also shows up what may be another problem (the HTTPS HandShakeFailed issue), but I'm less sure of that.

This repo contains the code that re-produces this problem and the Readme.md describes the reproduction. First of all I'm interested to see if you can reproduce it (others here in the office have reproduced the original problem on another variant of Linux and on Mac). Then I'd be interested in any advice on how to make further progress on tracking this down. Finally, I'm also interested to hear if you think that the HandshakeFailed errors described in the Readme.md are a bug (I actually think they are closely related to this TLS alert prepend issue).

Thanks!