douban / greenify Goto Github PK
View Code? Open in Web Editor NEWMake blocking C library work with gevent
License: BSD 3-Clause "New" or "Revised" License
greenify's People
Contributors
Stargazers
Watchers
Forkers
cmgs mckelvin e42s omerzimp thedrow southwolf abaelhe zebozhuang iamdafu txdywy sunchen009 newwayy bigr-lab pipamc alex8224 jinrenlab huachen0216 u20024804 chenbk85 irachex devinwang whynotyes dantangfan a740122 pypeng pengzhangdev journeyqiao gf0842wf junred windreamer zhu327 eternal1025 leonliu315 guojian huangkbaaron leehyong liurenjin layzerar hugoren junnplus mrstupidbird zuiwanting slucx wangjianze u2386 everpcpc dsuch ringsaturn mutalisk999 awesome-archive zheaoli nanramakr zak10 lexdene kentslaneygreenify's Issues
请教一下如何patch mysqlclient？
Oracle KPEDBG_HDL_PUSH_FCPTRMAX 11x Enter green_write
Hello,
I use greenify compiled from source code, commit 9bca4dc, along with SQLAlchemy, gevent and cx_Oracle - all newest versions from PyPI as of today.
I am trying to run 200 concurrent queries using a greenified version of cx_Oracle.
This works as long as I set the pool size not to exceed 11 connections. As soon as I set it to 12 or more, I get the following debug traceback from greenify and the process receives a SIGKILL.
As you can see below, there are exactly 11 Enter green_write entries without a corresponding return entry, such as "write, return 159, errno: 11."
You can also see a KPEDBG_HDL_PUSH_FCPTRMAX message which comes from Oracle - looking around, it is certainly to do with multi-threading but there are no details to be found what it is exactly about.
I wonder what makes greenify enter this code path where it has 11 "Enter green_write" messages and then it receives SIGKILL?
Thanks.
[greenify] [Fri Jan 29 19:51:41 2021] [19875] read, return 41, errno: 11
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] write, return 159, errno: 11
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_read
[greenify] [Fri Jan 29 19:51:41 2021] [19875] read, return -1, errno: 11
[greenify] [Fri Jan 29 19:51:41 2021] [19875] read, return 127, errno: 11
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] write, return 34, errno: 0
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_read
[greenify] [Fri Jan 29 19:51:41 2021] [19875] read, return -1, errno: 11
[greenify] [Fri Jan 29 19:51:41 2021] [19875] read, return 127, errno: 11
KPEDBG_HDL_PUSH_FCPTRMAX[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
[greenify] [Fri Jan 29 19:51:41 2021] [19875] Enter green_write
blocking function ~~~
Exception AssertionError: AssertionError('Impossible to call blocking function in the event loop callback',) in 'greenify.wait_gevent' ignored
某种大并发大流量的情况下会发生
AssertionError: Invalid switch into Greenlet
We got a few errors like that on prod - seems that the gevent switch failed. Among thousands of requests a few of them got the error below:
Traceback (most recent call last): File "greenify.pyx", line 71, in greenify.wait AssertionError: Invalid switch into <Greenlet at 0x7fd6ea869260: _handle_and_close_when_done(functools.partial(<bound method GeventWorker.handl, <bound method StreamServer.do_close of <StreamServ, (<gevent._socket3.socket at 0x7fd6eb29c9a0 object,)>: None Exception ignored in: 'greenify.wait_gevent' Traceback (most recent call last): File "greenify.pyx", line 71, in greenify.wait AssertionError: Invalid switch into <Greenlet at 0x7fd6ea869260: _handle_and_close_when_done(functools.partial(<bound method GeventWorker.handl, <bound method StreamServer.do_close of <StreamServ, (<gevent._socket3.socket at 0x7fd6eb29c9a0 object,)>: None
Python version: 3.9.10
Pyramid: 2.0
gevent: 21.12.0
gunicorn: 20.1.0
SQLAlchemy: 1.4.36
build wheel package with abi3
$ abi3audit --verbose dist/*.whl
greenify-0.4.1-cp37-abi3-linux_x86_64.whl:
greenify.cpython-38-x86_64-linux-gnu.so has non-ABI3 symbols
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Symbol ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ _Py_CheckRecursionLimit │
│ Py_OptimizeFlag │
└─────────────────────────┘
greenify-0.4.1-cp37-abi3-linux_x86_64.whl: 1 extensions scanned; 0 ABI
version mismatches and 2 ABI violations found
support more write/read system calls
$ readelf -Ws /usr/lib/libmysqlclient.so | ack write
13: 0000000000000000 0 FUNC GLOBAL DEFAULT UND SSL_write
23: 0000000000000000 0 FUNC GLOBAL DEFAULT UND write@GLIBC_2.2.5 (3)
109: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pwrite@GLIBC_2.2.5 (3)
134: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fwrite@GLIBC_2.2.5 (3)
pwrite和fwrite也需要greenify(pwrite, fwrite不会用在socket上，因此不需要)。类似地：
/usr/lib/libmemcached.so
/usr/lib/libmysqlclient.so
/usr/lib/libmysqlclient_r.so
至少这3个.so里含send/recv/write/read关键字的系统调用函数都需要支持。
$ readelf -Ws /usr/lib/libmemcached.so | ack "(read|write|send|recv)[a-z0-9]*@GLIB"
23: 0000000000000000 0 FUNC GLOBAL DEFAULT UND recv@GLIBC_2.2.5 (2)
43: 0000000000000000 0 FUNC GLOBAL DEFAULT UND sendmsg@GLIBC_2.2.5 (2)
57: 0000000000000000 0 FUNC GLOBAL DEFAULT UND send@GLIBC_2.2.5 (2)
71: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fwrite@GLIBC_2.2.5 (2)
$ readelf -Ws /usr/lib/libmysqlclient.so | ack "(read|write|send|recv)[a-z0-9]*@GLIB"
18: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fread@GLIBC_2.2.5 (3)
21: 0000000000000000 0 FUNC GLOBAL DEFAULT UND readlink@GLIBC_2.2.5 (3)
23: 0000000000000000 0 FUNC GLOBAL DEFAULT UND write@GLIBC_2.2.5 (3)
65: 0000000000000000 0 FUNC GLOBAL DEFAULT UND read@GLIBC_2.2.5 (3)
92: 0000000000000000 0 FUNC GLOBAL DEFAULT UND readdir@GLIBC_2.2.5 (3)
109: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pwrite@GLIBC_2.2.5 (3)
129: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pread@GLIBC_2.2.5 (3)
134: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fwrite@GLIBC_2.2.5 (3)
$ readelf -Ws /usr/lib/libmysqlclient_r.so | ack "(read|write|send|recv)[a-z0-9]*@GLIB"
9: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pwrite@GLIBC_2.2.5 (4)
39: 0000000000000000 0 FUNC GLOBAL DEFAULT UND read@GLIBC_2.2.5 (4)
80: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pread@GLIBC_2.2.5 (4)
93: 0000000000000000 0 FUNC GLOBAL DEFAULT UND readlink@GLIBC_2.2.5 (3)
101: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fread@GLIBC_2.2.5 (3)
146: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fwrite@GLIBC_2.2.5 (3)
155: 0000000000000000 0 FUNC GLOBAL DEFAULT UND write@GLIBC_2.2.5 (4)
需要补的几个调用是： pread, pwrite; 次优先级: readv, writev, recvfrom, recvmsg, sendto , sendfile
update:
线上在用3个.so中所有socket相关调用都有处理了。因此这个issue所需的其他几个调用只是有益补充。
Patch libmysqlclient.so core dump
Reproduce code as below
import greenify
greenify.greenify()
greenify.patch_lib("/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18")os: Ubuntu 14.04.3 LTS 64bit
greenify ver: 0.2
libmysqlclient version: 18
coredump.zip
Release to PyPi
There's no need to install from the repository unless you consider this project not ready for consumption.
Greenify
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.