worker continue die with gevent loop engine about uwsgi

[unbit]

does it happens on the first request or as soon as the server starts ?

[liujuncn]

As I observed, it happens on the first request starts.

After I switch to preforking mode, the web app can run normally. (BTW, the OS is 3.2.0-29-generic #46-Ubuntu 12.04 LTS server)

Same code will run normally with gevent loop engine under Centos 2.6.32-220.13.1.el6.x86_64.

[dmalenko]

I also experience similar crashes on first request after nobody touched the server for some time:

uwsgi(uwsgi_backtrace+0x25) [0x4452b5]
uwsgi(uwsgi_segfault+0x21) [0x445391]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f0f1ca4d4a0]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x3a) [0x7f0f1d14fe6a]
/usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f0f1d1507d7]
uwsgi(python_call+0x24) [0x451f94]
uwsgi(uwsgi_gevent_current_wsgi_req+0x1d) [0x46228d]
uwsgi(warn_pipe+0xa) [0x44228a]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f0f1ca4d4a0]
/lib/x86_64-linux-gnu/libpthread.so.0(write+0x2d) [0x7f0f1e4a5ccd]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(net_real_write+0x9a) [0x7f0f1656b87a]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(net_flush+0x1b) [0x7f0f1656bd4b]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(net_write_command+0x1a2) [0x7f0f1656bf12]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(cli_advanced_command+0x10c) [0x7f0f165691ac]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(mysql_close+0x59) [0x7f0f165674f9]
/opt/DAILY/local/lib/python2.7/site-packages/_mysql.so(+0x4ff4) [0x7f0f16a79ff4]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x53a5) [0x7f0f1d0a0845]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7f0f1d06b7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7f0f1d1341cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x131720) [0x7f0f1d140720]
/usr/lib/libpython2.7.so.1.0(+0x141f08) [0x7f0f1d150f08]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3fcd) [0x7f0f1d09f46d]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7f0f1d0a08c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7f0f1d0a08c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c8c0) [0x7f0f1d06b8c0]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2b2a) [0x7f0f1d09dfca]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7f0f1d0a08c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7f0f1d06b7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7f0f1d1341cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x131134) [0x7f0f1d140134]
/usr/lib/libpython2.7.so.1.0(PyObject_GetIter+0x3e) [0x7f0f1d144cae]
/usr/lib/libpython2.7.so.1.0(+0xdb7f0) [0x7f0f1d0ea7f0]
/usr/lib/libpython2.7.so.1.0(+0x81322) [0x7f0f1d090322]
/usr/lib/libpython2.7.so.1.0(+0x141f08) [0x7f0f1d150f08]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3fcd) [0x7f0f1d09f46d]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7f0f1d06b7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7f0f1d1341cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x10ee94) [0x7f0f1d11de94]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b33) [0x7f0f1d09efd3]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]

The system is also on Ubuntu 12.04 LTS using uWSGI 1.4.9 and gevent 1.0rc2.

[unbit]

how you are making mysql gevent friendly ?

[dmalenko]

You are right, I did not do anything specific to make mysql play nicely with gevent. So, I guess, my comment can be disregarded.

[prymitive]

Is this issue still valid? Does it work with 1.9?

[gilles]

Seems I'm running in a similar issue with 1.9, not sure it's the same though

*** HARAKIRI ON WORKER 4 (pid: 142962, try: 1) ***
*** backtrace of 142962 ***
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_backtrace+0x25) [0x453795]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(what_i_am_doing+0x30) [0x4538d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7fb4a3cd40b0]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7fb4a3d974d3]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(+0x22475) [0x7fb499be6475]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(ev_run+0x284) [0x7fb499bedde4]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(+0x2a909) [0x7fb499bee909]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5e75) [0x7fb4a43098d5]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff4a6) [0x7fb4a439d4a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa106d) [0x7fb4a433f06d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb4a43d0b37]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3913) [0x7fb4a1dba913]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3340) [0x7fb4a1dba340]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3d96) [0x7fb4a1dbad96]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb4a43d0b37]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa0499) [0x7fb4a433e499]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1bfe) [0x7fb4a430565e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4a00) [0x7fb4a4308460]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4a00) [0x7fb4a4308460]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff4a6) [0x7fb4a439d4a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa106d) [0x7fb4a433f06d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_CallMethod+0xea) [0x7fb4a42eaa3a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x46fe4a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_ignition+0xba) [0x453c0a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_start+0x1488) [0x4558e8]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x41c306]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb4a3cbeea5]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x41c331]
*** end of backtrace ***

[uwsgi]
socket = /tmp/app.socket
chown-socket = www-data
master = true
processes = 8
harakiri = 30
gevent = 5
gevent-monkey-patch = true
#threaded-logger = true => does not make any difference
stats = 127.0.0.1:9191
max-requests = 10000
wsgi-file = wsgi.py
lazy = true
#enable-threads = true => does not make any difference

[unbit]

no, it is a simple harakiri triggered because your app is blocked. You can get more infos about why your app was blocked adding the tracebacker (by default you only get the C traceback)