HTTPS requests varying runtime about hackney

[qrilka]

And httpc version of the same suite finishes in 5s

[benoitc]

@qrilka what do you mean by "HTTPS requests through proxy?"

[qrilka]

e.g. hackney:request(get, "https://some-url.com", [], [], [{proxy,{"proxy", 8080}}])

[benoitc]

I don't see any slowness right now... The code is pretty straightforward. The only cause I see right now is the use of TCP_KEEPALIVE when creating the connection. Also the socket is upgraded to handle ssl and reuse sessions .

Still investigating on it. Can you share the code of your mocking server?

[qrilka]

Looks like the problems come the line https://github.com/benoitc/hackney/blob/master/src/hackney_connect.erl#L192 - would you mind to explain why do we need to check ipv6 of the host requested through the proxy?
In our testcase we use fake domain names and so the slowdown comes from DNS query inside inet:getaddr.

[qrilka]

And also I don't undestand why don't I see the same slowdown for HTTP requests

[benoitc]

Why do you think there is no need to check if the host is under ipv6? What if you're using an ipv6 http proxy?

[qrilka]

And what's the point of that check? hackney connects to the proxy and not to the target host

[benoitc]

the host checked is the proxy one here....

[benoitc]

argh never mind. I will come with a patch since you still need to check if you're using an ipv6 host for the proxy

[qrilka]

And it looks like you don't pass any options when connecting to CONNECT proxy - https://github.com/benoitc/hackney/blob/master/src/hackney_http_connect.erl#L46

[qrilka]

So it looks like proxy and host are mixed up somewhere around https://github.com/benoitc/hackney/blob/master/src/hackney.erl#L673

[benoitc]

Yes. because you dont' want to there normally. Will figure how to handle better defaults.

Anyway the the patch I am thinking of is this one:

1. ProxyHost and ProxyPort should be passed as main arguments. And pass the target in options. Ie something like:

   hackney_http_connect: connect(ProxyHost, ProxyHost, Opts, Timeout).
   

   The connect host and port would contain the host and port we are connecting to. Imo the old logic is wrong.

2. Cache resolving. Once we detected that we connect using ipv6 to an host, we should cache it. So we don't do the test 2 times.

Thoughts?

[benoitc]

@qrilka indeed ....

[qrilka]

Totally agree with your idea.

[benoitc]

@qrilka can you test #99 and let me know if works for you ?

[qrilka]

Looks like you have an error in dealing with headers:

6> hackney:get("http://httpbin.org/get",[],[],[{proxy,{"localhost", 8080}}]).

=ERROR REPORT==== 18-Apr-2014::09:29:26 ===
proxy error: <<72,84,84,80,47,49,46,48,32,52,48,48,32,66,97,100,32,82,101,113,117,101,115,116,13,10,83,101,114,118,101,114,58,32,116,105,110,121,112,114,111,120,121,47,49,46,56,46,51,13,10,67,111,110,116,101,110,116,45,84,121,112,101,58,32,116,101,120,116,47,104,116,109,108,13,10,67,111,110,110,101,99,116,105,111,110,58,32,99,108,111,115,101,13,10,13,10>>
false

And in Wireshark I see

Hosthttpbin.org

for CONNECT request
Unfortunaltely I don't have time to dig further at this moment

[benoitc]

Sorry I broke the correct behaviour in d6200c3. It's fixed in d704dc3 :

1> hackney:start().
ok
2> {ok, _, _, Ref} = hackney:get("http://httpbin.org/get",[],[],[{proxy,{"localhost", 8888}}]).
{ok,200,
    [{<<"Via">>,<<"1.1 tinyproxy (tinyproxy/1.8.3)">>},
     {<<"Content-Length">>,<<"343">>},
     {<<"Date">>,<<"Fri, 18 Apr 2014 07:05:36 GMT">>},
     {<<"Access-Control-Allow-Origin">>,<<"*">>},
     {<<"Content-Type">>,<<"application/json">>},
     {<<"Server">>,<<"gunicorn/0.17.4">>}],
    #Ref<0.0.0.100>}
3> hackney:body(Ref).
{ok,<<"{\n  \"url\": \"http://httpbin.org/get\",\n  \"args\": {},\n  \"origin\": \"88.163.70.217\",\n  \"headers\": {\n    \"Host\": \""...>>}

[benoitc]

@qrilka I merged the PR with fixes in master. Everything seems OK here.

I also added a test to test multiple connections over proxy in f0dc107 . Not sure if you will be able to test it today but that would be cool so I can eventually close this issue.

Thanks for the report anyway!

[qrilka]

Works like a charm, thanks