ugreen + pycurl multi_curl: buffer overflow detected :-) about uwsgi

[unbit]

When you say you tried with 'greenlet', you mean you get the same error ?

[77cc33]

not really, lot of these errors on every request. and page doesn't load till the end (not always, some time just transferring forever)

epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): Bad file descriptor [core/event.c line 387]
epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): File exists [core/event.c line 272]
epoll_ctl(): Bad file descriptor [core/event.c line 387]

[unbit]

I will check-it in the next few days as i have an idea of revamp the async mode for some callback-based (like node.js) framework. In the last year the gevent plugin became the de-facto standard for non-blocking programming in uWSGI, so i suggest you to give a look at it (the current github version is extremely advanced). If you do not want to use gevent, then, just wait a bit ;)

[77cc33]

thank you

well I think that in my case ugreen will be much faster. just some internal feeling, but anyway will try to check this solution - thanks for the tip. and for sure will test your fix in a few days ))

[77cc33]

just tried gevent (not pycurl, but used urlib2 and it's much slower ;-((( )

even static yield 'abcd' with ugreen is much faster and CPU effective than the save with gevent. so hope that you'll fix ugreen as it's super fast thing.

[77cc33]

also do you have any idea of async mode with uwsgi and pypy? is there some solution so I could use uwsgi with PYPY in async mode?

[unbit]

i have made a bunch of modification to the current master. Can you try with the github code ?

[77cc33]

hi,

installed from master 1.4-dev, but nothing changed

DAMN ! worker 2 (pid: 9923) died, killed by signal 6 :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 9929)
*** buffer overflow detected ***: uwsgi terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f58f37c6807]
/lib/x86_64-linux-gnu/libc.so.6(+0x109700)[0x7f58f37c5700]
/lib/x86_64-linux-gnu/libc.so.6(+0x10a7be)[0x7f58f37c67be]
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4(curl_multi_fdset+0xa3)[0x7f57599dcb33]
/usr/lib/python2.7/dist-packages/pycurl.so(+0x6cfb)[0x7f5759c0acfb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x556f)[0x7f58f3d45a0f]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f58f3d10605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420)[0x7f58f3d458c0]
/usr/lib/libpython2.7.so.1.0(+0x5b76b)[0x7f58f3d0f76b]
/usr/lib/libpython2.7.so.1.0(PyIter_Next+0xb)[0x7f58f3d6808b]
uwsgi(uwsgi_response_subhandler_wsgi+0x55)[0x453425]
uwsgi(uwsgi_request_wsgi+0x3df)[0x45211f]
uwsgi(u_green_request+0x19)[0x466de9]
/lib/x86_64-linux-gnu/libc.so.6(+0x48650)[0x7f58f3704650]
======= Memory map: ========
00400000-0048a000 r-xp 00000000 08:01 103554630                          /usr/local/bin/uwsgi
00689000-0068a000 r--p 00089000 08:01 103554630                          /usr/local/bin/uwsgi
0068a000-00695000 rw-p 0008a000 08:01 103554630                          /usr/local/bin/uwsgi
00695000-0069a000 rw-p 00000000 00:00 0 
02500000-030b6000 rw-p 00000000 00:00 0                                  [heap]
030b6000-04dc5000 rw-p 00000000 00:00 0                                  [heap]
7f5758756000-7f5758817000 rw-p 00000000 00:00 0 
7f5758817000-7f575881a000 r-xp 00000000 08:01 110101494                  /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f575881a000-7f5758a19000 ---p 00003000 08:01 110101494                  /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f5758a19000-7f5758a1a000 r--p 00002000 08:01 110101494                  /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f5758a1a000-7f5758a1b000 rw-p 00003000 08:01 110101494                  /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f5758a1b000-7f5758a22000 r-xp 00000000 08:01 103548335                  /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f5758a22000-7f5758c21000 ---p 00007000 08:01 103548335                  /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f5758c21000-7f5758c22000 r--p 00006000 08:01 103548335                  /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f5758c22000-7f5758c23000 rw-p 00007000 08:01 103548335                  /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f5758c23000-7f5758c48000 r-xp 00000000 08:01 103548328                  /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f5758c48000-7f5758e48000 ---p 00025000 08:01 103548328                  /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f5758e48000-7f5758e49000 r--p 00025000 08:01 103548328                  /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f5758e49000-7f5758e4a000 rw-p 00026000 08:01 103548328                  /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f5758e4a000-7f5758e4b000 rw-p 00000000 00:00 0 
7f5758e4b000-7f5758f0f000 r-xp 00000000 08:01 103548333                  /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f5758f0f000-7f575910e000 ---p 000c4000 08:01 103548333                  /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f575910e000-7f5759118000 r--p 000c3000 08:01 103548333                  /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f5759118000-7f5759119000 rw-p 000cd000 08:01 103548333                  /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f5759119000-7f5759131000 r-xp 00000000 08:01 103552777                  /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f5759131000-7f5759331000 ---p 00018000 08:01 103552777                  /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f5759331000-7f5759332000 r--p 00018000 08:01 103552777                  /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f5759332000-7f5759333000 rw-p 00019000 08:01 103552777                  /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f5759333000-7f575936e000 r-xp 00000000 08:01 103548331                  /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f575936e000-7f575956e000 ---p 0003b000 08:01 103548331                  /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f575956e000-7f575956f000 r--p 0003b000 08:01 103548331                  /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f575956f000-7f5759571000 rw-p 0003c000 08:01 103548331                  /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f5759571000-7f5759578000 r-xp 00000000 08:01 110101662                  /lib/x86_64-linux-gnu/librt-2.15.so
7f5759578000-7f5759777000 ---p 00007000 08:01 110101662                  /lib/x86_64-linux-gnu/librt-2.15.so
7f5759777000-7f5759778000 r--p 00006000 08:01 110101662                  /lib/x86_64-linux-gnu/librt-2.15.so
7f5759778000-7f5759779000 rw-p 00007000 08:01 110101662                  /lib/x86_64-linux-gnu/librt-2.15.so
7f5759779000-7f57597ab000 r-xp 00000000 08:01 103552757                  /usr/lib/x86_64-linux-gnu/libidn.so.11.6.6
7f57597ab000-7f57599aa000 ---p 00032000 08:01 103552757                  /usr/lib/x86_64-linux-gnu/libidn.so.11.6.6
7f57599aa000-7f57599ab000 r--p 00031000 08:01 103552757                  /usr/lib/x86_64-linux-gnu/libidn.so.11.6.6
7f57599ab000-7f57599ac000 rw-p 00032000 08:01 103552757                  /usr/lib/x86_64-linux-gnu/libidn.so.11.6.6
7f57599ac000-7f5759a01000 r-xp 00000000 08:01 103552724                  /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
7f5759a01000-7f5759c01000 ---p 00055000 08:01 103552724                  /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
7f5759c01000-7f5759c03000 r--p 00055000 08:01 103552724                  /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
7f5759c03000-7f5759c04000 rw-p 00057000 08:01 103552724                  /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
7f5759c04000-7f5759c13000 r-xp 00000000 08:01 103553259                  /usr/lib/python2.7/dist-packages/pycurl.so
7f5759c13000-7f5759e12000 ---p 0000f000 08:01 103553259                  /usr/lib/python2.7/dist-packages/pycurl.so
7f5759e12000-7f5759e13000 r--p 0000e000 08:01 103553259                  /usr/lib/python2.7/dist-packages/pycurl.so
7f5759e13000-7f5759e15000 rw-p 0000f000 08:01 103553259                  /usr/lib/python2.7/dist-packages/pycurl.so
7f5759e15000-7f5759e18000 r-xp 00000000 08:01 103551885                  /usr/lib/python2.7/lib-dynload/_heapq.so
7f5759e18000-7f575a017000 ---p 00003000 08:01 103551885                  /usr/lib/python2.7/lib-dynload/_heapq.so
7f575a017000-7f575a018000 r--p 00002000 08:01 103551885                  /usr/lib/python2.7/lib-dynload/_heapq.so
7f575a018000-7f575a01a000 rw-p 00003000 08:01 103551885                  /usr/lib/python2.7/lib-dynload/_heapq.so
7f575a01a000-7f575a76e000 rw-p 00000000 00:00 0 
7f575a76e000-7f575f715000 rw-s 00000000 00:04 1310665                    /dev/zero (deleted)
7f575f715000-7f5764535000 rw-s 00000000 00:04 1310664                    /dev/zero (deleted)
7f5764535000-7f5764ef9000 rw-s 00000000 00:04 1310663                    /dev/zero (deleted)
7f5764ef9000-7f576642f000 rw-s 00000000 00:04 1310662                    /dev/zero (deleted)
7f576642f000-7f576643f000 rw-s 00000000 00:04 1310661                    /dev/zero (deleted)
7f576643f000-7f576b3e6000 rw-s 00000000 00:04 1310660                    /dev/zero (deleted)
7f576b3e6000-7f5770206000 rw-s 00000000 00:04 1310659                    /dev/zero (deleted)
7f5770206000-7f5770bca000 rw-s 00000000 00:04 1310658                    /dev/zero (deleted)
7f5770bca000-7f5772100000 rw-s 00000000 00:04 1310657                    /dev/zero (deleted)
7f5772100000-7f5772110000 rw-s 00000000 00:04 1310656                    /dev/zero (deleted)
7f5772110000-7f57770b7000 rw-s 00000000 00:04 1310655                    /dev/zero (deleted)
7f57770b7000-7f577bed7000 rw-s 00000000 00:04 1310654                    /dev/zero (deleted)
7f577bed7000-7f577c89b000 rw-s 00000000 00:04 1310653                    /dev/zero (deleted)
7f577c89b000-7f577ddd1000 rw-s 00000000 00:04 1310652                    /dev/zero (deleted)
7f577ddd1000-7f577dde1000 rw-s 00000000 00:04 1310651                    /dev/zero (deleted)
7f577dde1000-7f5782d88000 rw-s 00000000 00:04 1310650                    /dev/zero (deleted)
7f5782d88000-7f5787ba8000 rw-s 00000000 00:04 1310649                    /dev/zero (deleted)
7f5787ba8000-7f578856c000 rw-s 00000000 00:04 1310648                    /dev/zero (deleted)
7f578856c000-7f5789aa2000 rw-s 00000000 00:04 1310647                    /dev/zero (deleted)
7f5789aa2000-7f5789ab2000 rw-s 00000000 00:04 1310646                    /dev/zero (deleted)
7f5789ab2000-7f578ea59000 rw-s 00000000 00:04 1310645                    /dev/zero (deleted)
7f578ea59000-7f5793879000 rw-s 00000000 00:04 1310644                    /dev/zero (deleted)
7f5793879000-7f579423d000 rw-s 00000000 00:04 1310643                    /dev/zero (deleted)
7f579423d000-7f5795773000 rw-s 00000000 00:04 1310642                    /dev/zero (deleted)
7f5795773000-7f5795783000 rw-s 00000000 00:04 1310641                    /dev/zero (deleted)
7f5795783000-7f579a72a000 rw-s 00000000 00:04 1310640                    /dev/zero (deleted)
7f579a72a000-7f579f54a000 rw-s 00000000 00:04 1310639                    /dev/zero (deleted)
7f579f54a000-7f579ff0e000 rw-s 00000000 00:04 1310638                    /dev/zero (deleted)
7f579ff0e000-7f57a1444000 rw-s 00000000 00:04 1310637                    /dev/zero (deleted)
7f57a1444000-7f57a1454000 rw-s 00000000 00:04 1310636                    /dev/zero (deleted)
7f57a1454000-7f57a63fb000 rw-s 00000000 00:04 1310635                    /dev/zero (deleted)
7f57a63fb000-7f57ab21b000 rw-s 00000000 00:04 1310634                    /dev/zero (deleted)
7f57ab21b000-7f57abbdf000 rw-s 00000000 00:04 1310633                    /dev/zero (deleted)
7f57abbdf000-7f57ad115000 rw-s 00000000 00:04 1310632                    /dev/zero (deleted)
7f57ad115000-7f57ad125000 rw-s 00000000 00:04 1310631                    /dev/zero (deleted)
7f57ad125000-7f57ad327000 r--p 00000000 08:01 103547856                  /usr/lib/locale/locale-archive
7f57ad359000-7f57ad3db000 rw-p 00000000 00:00 0 
7f57ad3db000-7f57ad3dc000 ---p 00000000 00:00 0 
7f57ad3dc000-7f57ad41c000 rwxp 00000000 00:00 0 

[77cc33]

hi,

finally we solved this issue. this issue is related to curl function curl_multi_fdset and was fixed in libcurl 7.23.0 (curl_multi_fdset: avoid FD_SET out of bounds)

anyway thank you for your help! and please don't drop uGREEN mode, it's much faster and lighter than gevent.

[unbit]

uGreen is used in other components too so removing it is not expected ;)