cmullaparthi / ibrowse Goto Github PK

View Code? Open in Web Editor NEW

515.0 515.0 190.0 961 KB

Erlang HTTP client

License: Other

Erlang 99.58% Makefile 0.38% Shell 0.04%

ibrowse's People

Contributors

Stargazers

Watchers

Forkers

kocolosk ddossot abs algorist dreverri ohmforce fdmanana gersh boorad textendo halzy cloudant abhay jclopes kbrown miranetworks alexeyr tholschuh retnuh hyperthunk abanca dym mistagrooves cantstopgames nivertech helllamer bneil rcouch coolluck chuckdavies rflynn93 jbothma mmzeeman hudson2010 fictive-kin marutha archaelus jcrabtree vuleetu asconix-old wdshin chvanikoff chef alepharchives dougselph cloudian yuledd miguel-barreiro zenithght milindparikh soup-in-boots caox sportlane pib ecd egirshov mykook n1rvana cobusc zhoubaozhou flycodepl lywaterman jheeta stuart diegobill klaustrainer yuelejia divolgin paulperegud yunnet f355 i11 chinnurtb wudeng vagabond wonder-people phonebooth pombredanne namjae tigertext whisper1 id3as chaitanyapandit virtan sakishum yangcancai goldensurfer xpg lastcanal telemetricsolutions e42s sorawa axisofeval richbria benjaminplee markschmidt ipowow onlookerscuec wangxin39 kerasking

ibrowse's Issues

crash: parse_response/2

{'EXIT',
    {{function_clause,
         [{ibrowse_http_client,'-parse_response/2-lc$^0/1-0-',
              [<<>>],
              [{file,"src/ibrowse_http_client.erl"},
               {line,1012}]},
          {ibrowse_http_client,parse_response,2,
              [{file,"src/ibrowse_http_client.erl"},
               {line,1012}]},
          {ibrowse_http_client,handle_sock_data,2,
              [{file,"src/ibrowse_http_client.erl"},{line,285}]},
          {gen_server,handle_msg,5,
              [{file,"gen_server.erl"},{line,597}]},
          {proc_lib,init_p_do_apply,3,
              [{file,"proc_lib.erl"},{line,227}]}]},
     {gen_server,call,
         [<4646.13865.4>,
          {send_req,
              {{url,
                   "http://foohost:8010/foopath",
                   "foohost",8010,undefined,
                   undefined,
                   "/foopath",
                   http,hostname},
               [],head,<<>>,
               [{max_sessions,100},
                {max_pipeline_size,1000},
                {response_format,list}],
               30000}},
          30000]}}}}

sed -e s^%IBROWSE_VSN%^1.5.6^ ibrowse.app.src > ../ebin/ibrowse.app
/bin/sh: ibrowse.app.src: not found
/bin/sh: %IBROWSE_VSN%: not found
/bin/sh: 1.5.6: not found
make: *** [../ebin/ibrowse.app] Error 1
sed: command garbled: s

ets cur_sz + find_best_connection/2 bug

Hi there,
I think I found two more bugs...

To make it easier to debug, I added some error logging.

find_best_connection(Tid, Max_pipe) ->
    case ets:first(Tid) of
    {Cur_sz, Pid} when Cur_sz < Max_pipe ->
        ets:delete(Tid, {Cur_sz, Pid}),
        ets:insert(Tid, {{Cur_sz + 1, Pid}, []}),
        {ok, Pid};
    X ->
        error_logger:error_report([Tid, Max_pipe, X]),
        {error, retry_later}
    end.

Now that I have some visibility I can see that I get {error, retry_later} because of the gard (Cur_sz < Max_pipe). That's ok and expected...

=ERROR REPORT==== 19-May-2011::16:29:13 ===
{ibrowse,retry_later}

=ERROR REPORT==== 19-May-2011::16:29:13 ===
28699
1
1: <0.469.0>

But if I check the state of the process, I can see that it's idle and that the cur_pipeline_size = 0!

io:format("~p~n", [sys:get_status(list_to_pid("<0.469.0>"))]).
{status,<0.469.0>,
        {module,gen_server},
        [[{my_trace_flag,false},
          {ibrowse_trace_token,["69.90.7.149",58,"9950"]},
          {'$ancestors',[<0.463.0>,ibrowse,ibrowse_sup,<0.288.0>]},
          {http_prot_vsn,"HTTP/1.1"},
          {conn_close,"keep-alive"},
          {'$initial_call',{ibrowse_http_client,init,1}}],
         running,<0.463.0>,[],
         [{header,"Status for generic server <0.469.0>"},
          {data,[{"Status",running},
                 {"Parent",<0.463.0>},
                 {"Logged events",[]}]},
          {data,[{"State",
                  {state,"69.90.7.149",9950,10,#Ref<0.0.33.237942>,false,
                         undefined,[],false,#Port<0.4923>,false,[],
                         {[],[]},
                         undefined,idle,undefined,<<>>,0,0,[],undefined,
                         undefined,false,#Ref<0.0.33.237748>,undefined,false,
                         undefined,undefined,<<>>,undefined,false,28699,0,
                         {1305,836006,880134}}}]}]]}

That means that the ets state != process state...

And that's not all, eventually find_best_connection/2 will start picking that one broken connection over and over blocking all requests.

Any ideas?

send_req fails to pull down news.ycombinator.com with {error,retry_later}

I think ibrowse is choking because news.ycombinator.com lacks a content length (pg uses his own custom non-complaint web server for news.yc).

I tried to get it working for a few hours, but ended up using httpc:request instead.

Try for yourself: ibrowse:send_req("http://news.ycombinator.com/", [], get). Using v2.2.0 just returns {error, retry_later}.

Accept binary as header name / values

cc: @nivertech, @oribrost

Error when stopping ibrowe

Following error reported by Sergey Samokhi.

I found that when calling the following function:

test() ->
    ibrowse:start(),
    ibrowse:send_req("http://google.com/", [], get),
    ibrowse:stop().

ibrowse:stop() causes an error

=ERROR REPORT==== 31-Jan-2012::01:34:41 ===
** Generic server <0.160.0> terminating 
** Last message in was stop
** When Server state == {state,<0.158.0>,36883,"google.com",80,10,10,1,
                               undefined}
** Reason for termination == 
** {function_clause,[{ibrowse_lb,'-handle_call/3-fun-0-',
                                 [{<0.161.0>,0,0},[]],
                                 [{file,"src/ibrowse_lb.erl"},{line,115}]},
                     {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
                     {ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
                     {ets,foldl,3,[{file,"ets.erl"},{line,187}]},
                     {ibrowse_lb,handle_call,3,
                                 [{file,"src/ibrowse_lb.erl"},{line,115}]},
                     {gen_server,handle_msg,5,
                                 [{file,"gen_server.erl"},{line,578}]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,227}]}]}

=ERROR REPORT==== 31-Jan-2012::01:34:41 ===
** Generic server ibrowse terminating 
** Last message in was stop
** When Server state == {state,false}
** Reason for termination == 
** {{case_clause,
        {'EXIT',
            {{function_clause,
                 [{ibrowse_lb,'-handle_call/3-fun-0-',
                      [{<0.161.0>,0,0},[]],
                      [{file,"src/ibrowse_lb.erl"},{line,115}]},
                  {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
                  {ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
                  {ets,foldl,3,[{file,"ets.erl"},{line,187}]},
                  {ibrowse_lb,handle_call,3,
                      [{file,"src/ibrowse_lb.erl"},{line,115}]},
                  {gen_server,handle_msg,5,
                      [{file,"gen_server.erl"},{line,578}]},
                  {proc_lib,init_p_do_apply,3,
                      [{file,"proc_lib.erl"},{line,227}]}]},
             {gen_server,call,[<0.160.0>,stop]}}}},
    [{ibrowse_lb,stop,1,[{file,"src/ibrowse_lb.erl"},{line,93}]},
     {ibrowse,'-handle_call/3-fun-0-',2,[{file,"src/ibrowse.erl"},{line,820}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
     {ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
     {ets,foldl,3,[{file,"ets.erl"},{line,187}]},
     {ibrowse,handle_call,3,[{file,"src/ibrowse.erl"},{line,819}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,578}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
{'EXIT',
    {{{case_clause,
          {'EXIT',
              {{function_clause,
                   [{ibrowse_lb,'-handle_call/3-fun-0-',
                        [{<0.161.0>,0,0},[]],
                        [{file,"src/ibrowse_lb.erl"},{line,115}]},
                    {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
                    {ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
                    {ets,foldl,3,[{file,"ets.erl"},{line,187}]},
                    {ibrowse_lb,handle_call,3,
                        [{file,"src/ibrowse_lb.erl"},{line,115}]},
                    {gen_server,handle_msg,5,
                        [{file,"gen_server.erl"},{line,578}]},
                    {proc_lib,init_p_do_apply,3,
                        [{file,"proc_lib.erl"},{line,227}]}]},
               {gen_server,call,[<0.160.0>,stop]}}}},
      [{ibrowse_lb,stop,1,[{file,"src/ibrowse_lb.erl"},{line,93}]},
       {ibrowse,'-handle_call/3-fun-0-',2,
           [{file,"src/ibrowse.erl"},{line,820}]},
       {lists,foldl,3,[{file,"lists.erl"},{line,1197}]},
       {ets,do_foldl,4,[{file,"ets.erl"},{line,198}]},
       {ets,foldl,3,[{file,"ets.erl"},{line,187}]},
       {ibrowse,handle_call,3,
           [{file,"src/ibrowse.erl"},{line,819}]},
       {gen_server,handle_msg,5,
           [{file,"gen_server.erl"},{line,578}]},
       {proc_lib,init_p_do_apply,3,
           [{file,"proc_lib.erl"},{line,227}]}]},
     {gen_server,call,[ibrowse,stop]}}}

function_clause ibrowse_http_client:to_ascii/1

It is random and rare error. I've seen it in logs only several times on random web-pages and cannot reproduce it by hands.

But i think, it is fixable using following error info (look at "Reason for termination" function to_ascii/1 arguments at the end):

=ERROR REPORT==== 23-May-2012::16:02:09 ===
** Generic server <0.323.0> terminating
** Last message in was {tcp,#Port<0.7054>,
                            <<72,84,84,80,47,49,46,49,32,50,48,48,32,79,75,13,
                              10,83,101,114,118,101,114,58,32,110,103,105,110,
                              120,13,10,68,97,116,101,58,32,87,101,100,44,32,
                              50,51,32,77,97,121,32,50,48,49,50,32,49,50,58,
                              48,50,58,48,57,32,71,77,84,13,10,67,111,110,116,
                              101,110,116,45,84,121,112,101,58,32,116,101,120,
                              116,47,104,116,109,108,59,32,99,104,97,114,115,
                              101,116,61,119,105,110,100,111,119,115,45,49,50,
                              53,49,13,10,84,114,97,110,115,102,101,114,45,69,
                              110,99,111,100,105,110,103,58,32,99,104,117,110,
                              107,101,100,13,10,67,111,110,110,101,99,116,105,
                              111,110,58,32,107,101,101,112,45,97,108,105,118,
                              101,13,10,75,101,101,112,45,65,108,105,118,101,
                              58,32,116,105,109,101,111,117,116,61,50,48,13,
                              10,67,97,99,104,101,45,67,111,110,116,114,111,
                              108,58,32,109,97,120,45,97,103,101,61,49,50,48,
                              44,32,112,117,98,108,105,99,13,10,69,120,112,
                              105,114,101,115,58,32,87,101,100,44,32,50,51,32,
                              77,97,121,32,50,48,49,50,32,49,50,58,48,52,58,
                              48,57,32,71,77,84,13,10,67,111,110,116,101,110,
                              116,45,69,110,99,111,100,105,110,103,58,32,103,
                              122,105,112,13,10,13,10,31,139,8,0,0,0,0,0,0,3,
                              3,0,0,0,0,0,0,0,0,0>>}
** When Server state == {state,"newsru.com",80,20000,undefined,false,
                            undefined,[],false,#Port<0.7054>,false,[],
                            {[{request,
                                  {url,
                                      "http://newsru.com/sport/24apr2012/bonus.html",
                                      "newsru.com",80,undefined,undefined,
                                      "/sport/24apr2012/bonus.html",http,
                                      hostname},
                                  head,
                                  [{response_format,binary}],
                                  {<0.174.0>,#Ref<0.0.3.130630>},
                                  undefined,false,[],
                                  {1337,774529,505993},
                                  1048576,false,undefined,undefined,false,
                                  binary,#Ref<0.0.3.130631>}],
                             [{request,
                                  {url,"http://newsru.com","newsru.com",80,
                                      undefined,undefined,"/",http,hostname},
                                  head,
                                  [{response_format,binary}],
                                  {<0.161.0>,#Ref<0.0.3.126284>},
                                  undefined,false,[],
                                  {1337,774511,393636},
                                  1048576,false,undefined,undefined,false,
                                  binary,#Ref<0.0.3.126358>}]},
                            {request,
                                {url,"http://newsru.com","newsru.com",80,
                                    undefined,undefined,"/",http,hostname},
                                head,
                                [{response_format,binary}],
                                {<0.161.0>,#Ref<0.0.3.126284>},
                                undefined,false,[],
                                {1337,774511,393636},
                                1048576,false,undefined,undefined,false,
                                binary,#Ref<0.0.3.126358>},
                            get_body,"200",<<>>,0,0,
                            [{"Server","nginx"},
                             {"Date","Wed, 23 May 2012 12:01:51 GMT"},
                             {"Content-Type",
                              "text/html; charset=windows-1251"},
                             {"Transfer-Encoding","chunked"},
                             {"Connection","keep-alive"},
                             {"Keep-Alive","timeout=20"},
                             {"Expires","Wed, 23 May 2012 12:02:21 GMT"},
                             {"Cache-Control","max-age=30"},
                             {"Serv","ng1"},
                             {"Content-Encoding","gzip"}],
                            undefined,undefined,false,undefined,false,chunked,
                            chunk_start,
                            <<31,139,8,0,0,0,0,0,0,3,3,0,0,0,0,0,0,0,0,0>>,
                            undefined,false,61473,2,undefined}
** Reason for termination ==
** {function_clause,[{ibrowse_http_client,to_ascii,"K"},
                     {ibrowse_http_client,hexlist_to_integer,3},
                     {ibrowse_http_client,parse_11_response,2},
                     {ibrowse_http_client,handle_sock_data,2},
                     {gen_server,handle_msg,5},
                     {proc_lib,init_p_do_apply,3}]}

Version: latest git (2012-05-15).

Req Method: GET
Req URL: http://newsru.com/sport/24apr2012/bonus.html
Req Headers: [{"Accept-Encoding", "gzip"}]
Req Params: [{response_format, binary}]

What do you think about it? Buggy server or problems with chunked encoding?

Retry routing request after worker died with reason normal (connection closed by server after response sent - not an error)

Chandru, what do you think?

fdmanana@31cfa88

cheers

Inactivity timeout not cleared

Hi Chandru.

I found out that, with ibrowse 2.0.x, if I use a worker for several request and the first request had an activity timeout X specified in the options, the timer is never cleared, causing a subsequent request (after X milliseconds) to fail.

Patch at:

http://github.com/fdmanana/ibrowse/commit/072d4f865fb77609e6cd75639a36264fbfb19933

Cheers

bad dec2hex implementation

Hi Chandru. The commit message explains it all:

fdmanana@c4991ff

Found this issue when trying to send a chunk with size > 65535 bytes.
Replaced the implementation of dec2hex with a call to io_lib:format/2. Also we don't need the width anywhere in current ibrowse's code (and chunk headers don't apply restrictions to the size's width).

cheers

eunit is failing

Hello,

I have tried to run a make test on the master and I get this error:

[nrdufour 01:40 PM] ~/tmp/ibrowse $ make test
./rebar compile
==> ibrowse (compile)
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_test.erl
Compiled src/ibrowse_app.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse.erl
Compiled src/ibrowse_http_client.erl
./rebar eunit
==> ibrowse (eunit)
======================== EUnit ========================
module 'ibrowse'
module 'ibrowse_app'
module 'ibrowse_http_client'
module 'ibrowse_lb'
module 'ibrowse_lib'
  ibrowse_lib_tests: parse_urls_test_ (module 'ibrowse_lib_tests')...*failed*
::{assertMatch_failed,[{module,ibrowse_lib_tests},
                     {line,127},
                     {expression,"ibrowse_lib : parse_url ( \"http://[:1080:0:0:0:8:800:200C:417A:]:6000/foo?q=bar\" )"},
                     {pattern,"{ error , { invalid_ipv6_address , \":1080:0:0:0:8:800:200C:417A:\" } }"},
                     {value,{error,invalid_uri}}]}


  [done in 0.003 s]
module 'ibrowse_sup'
ibrowse_test: ue_test (module 'ibrowse_test')...[0.004 s] ok
module 'ibrowse_test_server'
=======================================================
  Failed: 1.  Skipped: 0.  Passed: 1.
ERROR: One or more eunit tests failed.
make: *** [test] Error 1

Any idea?

Git tag is set to 2.2.0, but ibrowse.app vsn says 2.1.3

Self explanatory.

websocket support

websocket is new and coming, it would be fantastic to see websocket support in ibrowse.

What't the licensing status?

Hello!

While packaging your application for Fedora, I stumbled over one issue - we didn't come in agreement about correct license tag for ibrowse - is it available under either BSD or LGPLv2+ license or both licenses' terms must be applied simultaneously (e.g. BSD and LGPLv2+)?

See this link for the details:
https://bugzilla.redhat.com/show_bug.cgi?id=581279#c5

ibrowse.hrl in src directory

Which means it isn't found in the standard search path which looks for things in include. If possible can this file be moved into the include directory.

[regression] ibrowse_lib:parse_url/1 error on ipv6 hostnames

After recent update, parse_url() became unable to properly parse ipv6 as hostname:

1> ibrowse_lib:parse_url("http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html").
{error,invalid_uri}

2> ibrowse_lib:parse_url("http://[1080:0:0:0:8:800:200C:417A]/index.html").                
{error,invalid_uri}

3> ibrowse_lib:parse_url("http://[::192.9.5.5]/ipng").                     
{error,invalid_uri}

Example URLs is taken from http://www.ietf.org/rfc/rfc2732.txt

Regression is not reproduceable before this commit: 26d34d5

Adding makefile target to run the tests

Hi Chandru,

Could you add a target to run the tests?

http://github.com/fdmanana/ibrowse/compare/makefile_tests

I think this is fine and useful.

cheers

Timeouts in ibrowse:get_lb_pid and ibrowse_lb:spawn_connection

I am seeing errors in my log like

** Generic server my_server terminating
** Last message in was timeout

** When Server state == ...
** Reason for termination == 
** {timeout,
       {gen_server,call,
           [ibrowse,
            {get_lb_pid,
                {url,
                    "http://192.168.1.103:80/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
                    "192.168.1.103",80,undefined,undefined,
                    "/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
                    http,ipv4_address}}]}}

and

** Reason for termination == 
** {timeout,
       {gen_server,call,
           [<0.118.0>,
            {spawn_connection,
                {url,
                    "http://192.168.1.103:80/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
                    "192.168.1.103",80,undefined,undefined,
                    "/greengo/medialib/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi",
                    http,ipv4_address},
                10,10,
                {[],false}}]}}

after calling ibrowse:send_req/5 with timeout infinity.

connections are never closed

ibrowse never timesout on a open connection if the server never closes the connection.
This will cause an application that connects to several different domains to eventually exhaust all file descriptors.

I suggest the use of some sort of argument to explicitly pass a timeout in seconds or infinity to a send_req.
The connection should be closed after the timeout has expired.

atom pollution

I noticed that ibrowse was polluting the virtualmachine with atoms when passed bogus URLs.
The functions ibrowse_lib:parse_url calls list_to_atom to get the URL protocol but if it's not a well formed URL it will pollute the node with useless atoms.

I see 2 possible solutions:

keep the protocol as a string
check if it's a supported protocol before the list_to_atom call ["http", "https"]

error: function_clause ibrowse_http_client make request

Hi,

I encountered today an error when doing an HTTPS POST Request and I don't really know why.
The following error is printed:

{'EXIT',
    {{function_clause,
    [{ibrowse_http_client,
      '-make_request/8-lc$^0/1-0-',
      [{"Host",
        ["example.com",
         58,
         "443"]},
       #Fun<ibrowse_http_client.1.70135144>]},
     {ibrowse_http_client,
      '-make_request/8-lc$^0/1-0-',
      2},
     {ibrowse_http_client,
      make_request,
      8},
     {ibrowse_http_client,
      send_req_1,
      8},
     {gen_server,
      handle_msg,
      5},
     {proc_lib,
      init_p_do_apply,
      3}]},
    {gen_server,
    call,
    [<0.543.0>,
     {send_req,
      {{url,
        "https://example.com:443/someresource",
        "example.com",
        443,
        undefined,
        undefined,
        "/someresource",
        https,
        hostname},
       [{"Content-Type",
         "application/json;charset=UTF-8"},
        {"Authorization",
         "SomeAuthorization"},
        {"User-Agent",
         "SomeUserAgent"},
        {"Accept",
         "application/json;charset=UTF-8"}|
        {"Host",
         ["example.com",
          58,
          "443"]}],
       post,
       <<"{\"badge\":1}">>,
       [],
       30000}},
     30000]}}}

It would be nice if someone could shed some light on this.
Of course ibrowse:start() and ssl:start() has been called.

Code is the following:

handle_cast({example_api, UserId, Count}, State) ->
    Url = lists:flatten(io_lib:format("~s/~s", [State#state.url, UserId])),
    Headers = State#state.headers,
    Body = lists:flatten(mochijson:encode({struct, [{badge, Count}]})),
    ?DEBUG("Body: ~p", [Body]),
    ?DEBUG("Headers: ~p", [Headers]),
    ?DEBUG("Url: ~p", [Url]),

    case ibrowse:send_req(Url, Headers, post, Body) of
        {ok, Status, ResponseHeaders, ResponseBody} ->
            case Status of
                "202" ->
                    {noreply, State};
                "400" ->
                    ?ERROR_MSG("Got a 400 BAD Request.~nResponseHeaders: ~p~nResponseBody: ~p", 
                        [ResponseHeaders, ResponseBody]),
                    {noreply, State}
            end;
        {error, Reason} ->
            ?ERROR_MSG("Error, reason: ~p", [Reason]),
            {noreply, State}
    end.

Avoid exception when trying to close a link worker

Please, read the comment of the following commit:

http://github.com/fdmanana/ibrowse/commit/8b963892acbdb7d9e42ef6f59ff76717c93a0742

cheers

ibrowse leaks processes / memory

ibrowse never seems to clean up the gen_servers it creates, making it difficult to use for crawling. There could be an option to limit the maximum number of load balancers / clients it creates.

progress callback

Hi,
I am working on one project, which need show the download speed during a interval repeatly. I know i can use {stream_to, self()} to handle the write file part. but it is not efficient engouth that data passed as message and write to file.

So is it possible to add one option which is used to get the download speed infomation when use save_response_to_file for send_req/send_req_direct?

Thank you.

sub binary use is non-optimal

After compiling with ERL_COMPILER_OPTIONS=bin_opt_info set for binary use warnings, this was printed.

It might be good to fix these.

src/ibrowse_http_client.erl:974: Warning: NOT OPTIMIZED: called function chunk_request_body/3 does not begin with a suitable binary matching instruction
src/ibrowse_http_client.erl:999: Warning: variable 'State' is unused
src/ibrowse_http_client.erl:1494: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1497: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1509: Warning: INFO: using a matched out sub binary will prevent delayed sub binary optimization
src/ibrowse_http_client.erl:1509: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1513: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1516: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1524: Warning: OPTIMIZED: creation of sub binary delayed
src/ibrowse_http_client.erl:1530: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1544: Warning: INFO: using a matched out sub binary will prevent delayed sub binary optimization
src/ibrowse_http_client.erl:1544: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1548: Warning: NOT OPTIMIZED: sub binary is used or returned
src/ibrowse_http_client.erl:1559: Warning: OPTIMIZED: creation of sub binary delayed
src/ibrowse_http_client.erl:1634: Warning: OPTIMIZED: creation of sub binary delayed

ssl certificate alternative names

Does ibrowse support ssl certificate alternative names?
If I run
ibrowse:send_req("https://github.com", [], get).
it fails with
{error,{conn_failed,{error,esslconnect}}}

httpc:request("https://github.com").
on the other hand works fine (but is it checking the cert in any way???)

I'm not sure if the issue is alternative names, but it looks like it. Here's the discussion:
https://github.com/blog/738-sidejack-prevention-phase-2-ssl-everywhere#comment-9002

Dead connections are not being restarted...

Hi,
I've been experiencing a weird bug where some of the ibrowse connections die and are never restarted.... It usually take about 24 hours at 2500 QPS to kill all of the 100 connections (max_sessions).

Here is what my request looks like:

Body = "{\"name\": \"Bob\"}",
Headers = [{"Connection", "Keep-Alive"},
           {"Content-Type", "application/json"}],
Options = [{connect_timeout, ?CONNECTION_TIMEOUT},
           {max_sessions, 100},
           {max_pipeline_size, 1},
           {stream_to, self()}],

ibrowse:send_req("69.90.7.1:9950", Headers, post, Body, Options, 50)

If you check the status everything looks good....

ibrowse:show_dest_status().
Server:port                              | ETS   | Num conns  | LB Pid
================================================================================
69.90.7.1:9950                         | 28699 | 100   | <0.658.0>
ok

But if you check netstat, you'll see that there's only 76 sockets open and not 100!

netstat -n | grep 69.90.7.1:9950 | grep ESTABLISHED | wc -l
76

Now If you dig into the LB process, everything looks good...

process_info(list_to_pid("<0.658.0>")).
[{current_function,{gen_server,loop,6}},
 {initial_call,{proc_lib,init_p,5}},
 {status,waiting},
 {message_queue_len,0},
 {messages,[]},
 {links,[<0.7547.2582>,<0.9962.3775>,<0.30613.3784>,
         <0.32157.3784>,<0.1168.3785>,<0.1521.3785>,<0.1522.3785>,
         <0.1525.3785>,<0.1176.3785>,<0.32166.3784>,<0.32170.3784>,
         <0.1166.3785>,<0.32167.3784>,<0.32162.3784>,<0.32163.3784>,
         <0.32158.3784>,<0.30624.3784>,<0.32154.3784>,<0.32155.3784>,
         <0.30625.3784>,<0.30622.3784>|...]},
 {dictionary,[{my_trace_flag,false},
              {ibrowse_trace_token,["LB: ","69.90.7.1",58,"9950"]},
              {'$ancestors',[ibrowse,ibrowse_sup,<0.288.0>]},
              {'$initial_call',{ibrowse_lb,init,1}}]},
 {trap_exit,true},
 {error_handler,error_handler},
 {priority,normal},
 {group_leader,<0.287.0>},
 {total_heap_size,3571},
 {heap_size,2584},
 {stack_size,9},
 {reductions,1530637035},
 {garbage_collection,[{min_bin_vheap_size,46368},
                      {min_heap_size,233},
                      {fullsweep_after,65535},
                      {minor_gcs,16917}]},
 {suspending,[]}]

But when you check some of it's children... you can see that there's no link to a #Port!

 process_info(list_to_pid("<0.7547.2582>")).
 [{current_function,{gen_server,loop,6}},
  {initial_call,{proc_lib,init_p,5}},
  {status,waiting},
  {message_queue_len,0},
  {messages,[]},
  {links,[<0.658.0>]},
  {dictionary,[{my_trace_flag,false},
               {ibrowse_trace_token,["69.90.7.1",58,"9950"]},
               {'$ancestors',[<0.658.0>,ibrowse,ibrowse_sup,<0.288.0>]},
               {'$initial_call',{ibrowse_http_client,init,1}}]},
  {trap_exit,false},
  {error_handler,error_handler},
  {priority,normal},
  {group_leader,<0.287.0>},
  {total_heap_size,233},
  {heap_size,233},
  {stack_size,9},
  {reductions,42},
  {garbage_collection,[{min_bin_vheap_size,46368},
                       {min_heap_size,233},
                       {fullsweep_after,65535},
                       {minor_gcs,0}]},
  {suspending,[]}]

Any ideas?

Bug detecting existing request headers

In ibrowse_http_client:make_request/7 there's a call to ibrowse_lib:get_value(content_length, Headers, false) and ibrowse_lib:get_value(transfer_encoding, Headers, false). get_value does not attempt to check if the header exists as a string which ends up causing things like having multiple Content-Length headers set.

http://github.com/cmullaparthi/ibrowse/blob/master/src/ibrowse_http_client.erl#L806

Request

Is it possible to add debug_info in the list of compiler options?

diff --git a/src/Makefile b/src/Makefile
index 145c0a2..9c1bf63 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -13,7 +13,7 @@ INCLUDE_DIRS = -I./
 
 ERLC ?= erlc
 ERLC_EMULATOR ?= erl -boot start_clean
-COMPILER_OPTIONS = -W +warn_unused_vars +nowarn_shadow_vars +warn_unused_import
+COMPILER_OPTIONS = -W +debug_info +warn_unused_vars +nowarn_shadow_vars +warn_unused_import

Also, you may want to run dialyzer with the -Wunmatched_returns option and possibly correct these warnings. I can help, if needed.

Kostis

It looks like inactivity_timeout option doesn't always work

After starting a download with

25/Apr/2011 12:59:24.091  Debug: ibrowse options: [{save_response_to_file,
                       {append,
                           "/mware/working_dir/media/video/7e6fb725-36e6-47ac-870e-2876571eb024.avi"}},
                  {socket_options,[{keepalive,true}]},
                  {inactivity_timeout,180000},
                  {connect_timeout,180000}]

(and infinity timeout), there is more than 3 minutes of inactivity but the connection isn't broken.

Set proper git tags for versions.

This greatly helps in packaging ibrowse. GitHub automatically creates download links for each tag and there is no need to dig into sources to find exact commit for particular version.

build don't work when launched from a script

When I try to build ibrowse from commande line, it just works. When I try to automated it with a script, the %IBROWSE_VSN% is not resolved.
Why don't you use rebar?

Issues when streaming a chunk encoded response

Hi Chandru,

The following patch (I polished it a bit to reduce code size) fixes issues (ibrowse hanging or timing out) when streaming chunked encoded responses:

http://github.com/fdmanana/ibrowse/commit/e9ce3f9e999cc10bb971091b8fee18edfd106f58

Let me know if it's fine for you.
cheers

Automatically do chunked transfer-encoding when the body is a function

Hi Chandru,

The following patch adds support for adding chunked transfer-encoding when streaming requests with a user supplied function:

http://github.com/fdmanana/ibrowse/commit/6a6852c8b59de79a52c2c00d57909c006a7e635a

Set Content-Length for empty PUTs and POSTs and use iolist_size(Body) because body can be an iolist

Hi Chandru,

what do you think?

fdmanana@9611896

cheers

base64 encoding/decoding code

Hi Chandrashekhar,

First of all, thanks for ibrowse. A very useful, and well done, library.
We use it in CouchDB, and I was about to apply the following patch to the ibrowse version shipped with CouchDB:

http://github.com/fdmanana/ibrowse/commit/1bdc1185c67c3845969996ad1b3824be9b4afa8d

Do you think you can apply it as well to standard ibrowse? We plan to bump our version to the latest soon, so it would be great to have our version as close to yours as possible.

Thanks, and keep up the good work.

ibrowse_lib:parse_url/1 inconsistent behaviour on bad input

I've been using ibrowse_lib to parse urls and have found that the behaviour is inconsistent on bad input.

1> ibrowse_lib:parse_url("http://domain:not_a_port").
** exception error: bad argument
     in function  list_to_integer/1
        called as list_to_integer("not_a_port")
     in call from ibrowse_lib:parse_url/4
     in call from ibrowse_lib:parse_url/1
2> ibrowse_lib:parse_url("http://domain:80:").
** exception error: bad argument
     in function  list_to_integer/1
        called as list_to_integer("80:")
     in call from ibrowse_lib:parse_url/4
     in call from ibrowse_lib:parse_url/1
3> ibrowse_lib:parse_url("http:/domain:80:"). 
{invalid_uri_2,get_protocol,
               {url,"http:/domain:80:",undefined,undefined,undefined,
                    undefined,undefined,undefined,undefined},
               ":08:niamod/:ptth"}

As I see it the function should either always crash, or always return some sort of error message. Not chose randomly depending on the reason the input is wrong.

Accept iolist() as a request body

cc: @oribrost , @nivertech

Oauth 2.0 support

It would be nice to have Oauth 2. support in Erlang. Is ibowse good place for that?

https://github.com/tim/erlang-oauth/issues/4#comment_1218849

Regards
Matteo
http://www.redaelli.org/matteo/

Unable to make https request from behind proxy

If I try to make and https request from behind a proxy I always get {error,conn_failed}. The same request over http succeeds.

If I try to set ssl_options like this {ssl_options, [{ssl_imp, new}]} I instead get the exception

=ERROR REPORT==== 20-Apr-2010::14:44:57 ===
SSL: hello: ./ssl_record.erl:292:Fatal error: record_overflow
{error,conn_failed}

Is it supported to use SSL from behind a proxy?

302 HTTP

is there any options for urls redirection 302 "Moved Temporally".. thank's

Document default timeout of 30 seconds for ibrowse:send_req/3,4,5

I think infinity is the more expected value.

save_response_to_file regression in 2.1.3

There was a regression in save_response_to_file option in v2.1.3. Instead of returning a response in the non-2xx case it throws a badmatch. This exact request works fine in 2.1.2.

7> ibrowse:send_req("http://www.google.com/tsdfasd", [], get, [], [{save_response_to_file, "response"}], 10000).
{error,{'EXIT',{{{badmatch,{error,badarg}},
                 [{ibrowse_http_client,handle_response,2},
                  {ibrowse_http_client,parse_11_response,2},
                  {ibrowse_http_client,handle_sock_data,2},
                  {gen_server,handle_msg,5},
                  {proc_lib,init_p_do_apply,3}]},
                {gen_server,call,
                            [<0.47.0>,
                             {send_req,{{url,"http://www.google.com/tsdfasd",
                                             "www.google.com",80,undefined,undefined,"/tsdfasd",http},
                                        [],get,<<>>,
                                        [{save_response_to_file,"response"}],
                                        10000}},
                             10000]}}}}
8> 
=ERROR REPORT==== 31-Jan-2011::10:35:49 ===
** Generic server <0.47.0> terminating 
** Last message in was {tcp,#Port<0.706>,
                            <<"=0 cellspacing=0><tr><td bgcolor=\"#3366cc\"><img alt=\"\" width=1 height=4></td></tr></table>\n</body></html>\n">>}
** When Server state == {state,"www.google.com",80,10000,undefined,false,
                            undefined,[],false,#Port<0.706>,false,[],
                            {[{request,
                                  {url,"http://www.google.com/tsdfasd",
                                      "www.google.com",80,undefined,undefined,
                                      "/tsdfasd",http},
                                  get,
                                  [{save_response_to_file,"response"}],
                                  {<0.33.0>,#Ref<0.0.0.73>},
                                  undefined,false,[],
                                  {1296,488149,148952},
                                  1048576,"response",undefined,undefined,
                                  false,list}],
                             []},
                            {request,
                                {url,"http://www.google.com/tsdfasd",
                                    "www.google.com",80,undefined,undefined,
                                    "/tsdfasd",http},
                                get,
                                [{save_response_to_file,"response"}],
                                {<0.33.0>,#Ref<0.0.0.73>},
                                undefined,false,[],
                                {1296,488149,148952},
                                1048576,"response",undefined,undefined,false,
                                list},
                            get_body,"404",
                            <<"\n\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>404 Not Found</title>\n<style><!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--></style>\n<script><!--\nvar rc=404;\n//-->\n</script>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<table border=0 cellpadding=2 cellspacing=0 width=100%><tr><td rowspan=3 width=1% nowrap>\n<b><font face=times color=#0039b6 size=10>G</font><font face=times color=#c41200 size=10>o</font><font face=times color=#f3c518 size=10>o</font><font face=times color=#0039b6 size=10>g</font><font face=times color=#30a72f size=10>l</font><font face=times color=#c41200 size=10>e</font>&nbsp;&nbsp;</b>\n<td>&nbsp;</td></tr>\n<tr><td bgcolor=\"#3366cc\"><font face=arial,sans-serif color=\"#ffffff\"><b>Error</b></td></tr>\n<tr><td>&nbsp;</td></tr></table>\n<blockquote>\n<H1>Not Found</H1>\nThe requested URL <code>/tsdfasd</code> was not found on this server.\n\n<p>\n</blockquote>\n<table width=100% cellpadding">>,
                            1255,0,
                            [{"Content-Type","text/html; charset=UTF-8"},
                             {"X-Content-Type-Options","nosniff"},
                             {"Date","Mon, 31 Jan 2011 15:35:49 GMT"},
                             {"Server","sffe"},
                             {"Content-Length","1361"},
                             {"X-XSS-Protection","1; mode=block"}],
                            undefined,undefined,false,#Ref<0.0.0.77>,1361,
                            false,undefined,undefined,<<>>,undefined,false,
                            32787,1,undefined}
** Reason for termination == 
** {{badmatch,{error,badarg}},
    [{ibrowse_http_client,handle_response,2},
     {ibrowse_http_client,parse_11_response,2},
     {ibrowse_http_client,handle_sock_data,2},
     {gen_server,handle_msg,5},
     {proc_lib,init_p_do_apply,3}]}

src/ibrowse_http_client.erl:999: Warning: variable 'State' is unused

Seems a little odd to have a single compilation warning.

deps/ibrowse/src/ibrowse_http_client.erl:999: Warning: variable 'State' is unused

[ibrowse ((v4.0.1))]$ git log -n 1
commit 9b8b042
Author: Chandrashekhar Mullaparthi [email protected]
Date: Tue Aug 7 07:05:37 2012 +0100

Fixed issue #67 with eyes wide open

Broken version in ibrowse.app.src

{vsn, git} is not being correctly replaced when used as a rebar dependency eg

{deps, [{ibrowse, "2.2.*", {git, "git://github.com/cmullaparthi/ibrowse", "HEAD"}}]}

fails with

Uncaught error in rebar_core: {'EXIT',
{badarg,
[{re,run,[git,"2.2.0",[{capture,none}]]},
{rebar_deps,is_app_available,3},
{rebar_deps,find_deps,3},
{rebar_deps,preprocess,2},
{rebar_core,acc_modules,5},
{rebar_core,process_dir,4},
{rebar_core,process_commands,1},
{rebar,main,1}]}}
make: *** [deps] Error 1

dialyzer unmatched_returns warnings

I haven't checked the details but there are more than a few dialyzer warnings that might be of interest.

$ dialyzer --plt ~/.dialyzer_plt.R14B03 -Wunmatched_returns -r ./

ibrowse_http_client.erl:191: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:294: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:318: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:331: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:348: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:570: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:573: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:576: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:692: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:775: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:783: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_http_client.erl:1374: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:79: Expression produces a value of type atom() | tid(), but this value is unmatched
ibrowse_test.erl:80: Expression produces a value of type atom() | tid(), but this value is unmatched
ibrowse_test.erl:81: Expression produces a value of type atom() | tid(), but this value is unmatched
ibrowse_test.erl:145: Expression produces a value of type 'ok' | integer(), but this value is unmatched
ibrowse_test.erl:189: Expression produces a value of type 'ok' | {'error',atom()}, but this value is unmatched
ibrowse_test.erl:233: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:234: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:235: Expression produces a value of type 'ok' | {'error',}, but this value is unmatched
ibrowse_test.erl:237: Expression produces a value of type 'ignore' | {'error',} | {'ok',pid()}, but this value is unmatched
ibrowse_test.erl:379: Expression produces a value of type 'ok' | {'error','unknown_req_id'}, but this value is unmatched
ibrowse_test.erl:386: Expression produces a value of type 'ok' | {'error','unknown_req_id'}, but this value is unmatched
Unknown functions:
ibrowse_test_server:start_server/2
done in 0m24.47s
done (warnings were emitted)

ibrowse.app contains old version

vsn.mk != ibrowse.app

The {save_response_to_file, filename()} option is ignored

I tried requesting

ibrowse:send_req("http://www.google.com", [], get, [], [{save_response_to_file, "/tmp/foo.txt"}]).

and expected a file /tmp/foo.txt but instead the response body was returned. {save_response_to_file, true} worked just fine.

Receiving unexpected message from ibrowse process

When I call

ibrowse:send_req(Url, [], get, [], 
             [{save_response_to_file,
                  "/mware/working_dir/media/video/7eb8971f-0527-4c6c-af93-11aa0afb0c1a.avi"},
              {socket_options,[{keepalive,true}]},
              {inactivity_timeout,180000},
              {connect_timeout,180000}])

and the request times out, it returns {error,req_timedout}. But then the process which called send_req also receives a message {#Ref<0.0.0.821>,{error,req_timedout}}, which is not expected, since there is no {stream_to, self()} option.

tcp_error handling in ibrowse_http_client

Hi Chandru,

I stumbled into this when browsing the source on github:

tcp_error comes with 2 parameters: {tcp_error, Socket, Reason}.

handle_info({tcp_error, _Sock}, State) in ibrowse_http_client probably won't match it.

I suspect it might leave dead Pids in ETS tables.

BR,
Ferenc

cmullaparthi / ibrowse Goto Github PK

ibrowse's People

Contributors

Stargazers

Watchers

Forkers

ibrowse's Issues

Recommend Projects

Recommend Topics

Recommend Org