ethereum / asyncio-run-in-process Goto Github PK
View Code? Open in Web Editor NEWA simple asyncio friendly replacement for multiprocessing to run coroutines in a separate process.
License: MIT License
A simple asyncio friendly replacement for multiprocessing to run coroutines in a separate process.
License: MIT License
Custom exceptions whose constructor require multiple arguments cannot be pickled (https://bugs.python.org/issue32696) and that currently cause us to crash
One example of such exception is rlp.exceptions.SerializationError
Custom exceptions whose constructor require multiple arguments cannot be pickled (https://bugs.python.org/issue32696) and that currently cause us to crash
One example of such exception is rlp.exceptions.SerializationError
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.connect_read_pipe
Currently, the parent process uses blocking IO and asyncio.run_in_executor
to read/write to the pipes used to communicate with the child process. It appears that we can use the loop.connect_read_pipe
to construct the appropriate StreamReader
and StreamWriter
instances to make this IO non-blocking and remove the need for a thread based executor.
A few tests related to SIGINT
and process killing are failing on my machine. I assume this is the cause of ethereum/trinity#1507.
The following two test cases are failing:
tests/core/test_open_in_process.py::test_open_proc_SIGINT_can_be_handled FAILED [ 15%]
tests/core/test_open_in_process.py::test_open_proc_SIGINT_can_be_ignored FAILED [ 18%]
Sometimes, they fail immediately, sometimes they block indefinitely and I have to kill them manually.
================================================================================== FAILURES ==================================================================================
____________________________________________________________________ test_open_proc_SIGINT_can_be_handled ____________________________________________________________________
@pytest.mark.asyncio
async def test_open_proc_SIGINT_can_be_handled():
async def do_sleep_forever():
try:
while True:
await asyncio.sleep(0)
except KeyboardInterrupt:
return 9999
async with open_in_process(do_sleep_forever) as proc:
proc.send_signal(signal.SIGINT)
> assert proc.returncode == 0
E assert 2 == 0
E -2
E +0
do_sleep_forever = <function test_open_proc_SIGINT_can_be_handled.<locals>.do_sleep_forever at 0x7f7cc9b84c10>
proc = <asyncio_run_in_process.process.Process object at 0x7f7cc9b34430>
tests/core/test_open_in_process.py:58: AssertionError
____________________________________________________________________ test_open_proc_SIGINT_can_be_ignored ____________________________________________________________________
@pytest.mark.asyncio
async def test_open_proc_SIGINT_can_be_ignored():
async def do_sleep_forever():
try:
while True:
await asyncio.sleep(0)
except KeyboardInterrupt:
# silence the first SIGINT
pass
try:
while True:
await asyncio.sleep(0)
except KeyboardInterrupt:
return 9999
async with open_in_process(do_sleep_forever) as proc:
proc.send_signal(signal.SIGINT)
await asyncio.sleep(0.01)
> proc.send_signal(signal.SIGINT)
do_sleep_forever = <function test_open_proc_SIGINT_can_be_ignored.<locals>.do_sleep_forever at 0x7f7cc9b849d0>
proc = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>
tests/core/test_open_in_process.py:81:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>, signum = <Signals.SIGINT: 2>
def send_signal(self, signum: signal.Signals) -> None:
"""
Issues the provided signal to the process.
"""
> os.kill(self.pid, signum.value)
E ProcessLookupError: [Errno 3] No such process
self = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>
signum = <Signals.SIGINT: 2>
asyncio_run_in_process/process.py:277: ProcessLookupError
$ python -m eth_utils
Python version:
3.8.1 (default, Dec 21 2019, 20:57:38)
[GCC 9.2.0]
Operating System: Linux-4.19.91-1-MANJARO-x86_64-with-glibc2.2.5
pip freeze result:
alabaster==0.7.12
apipkg==1.5
argh==0.26.2
async-generator==1.10
-e [email protected]:ethereum/asyncio-run-in-process.git@74f11319739c64c0fe8ff7c2aebee3ecf5a125e8#egg=asyncio_run_in_process
atomicwrites==1.3.0
attrs==19.3.0
Babel==2.8.0
backcall==0.1.0
bleach==3.1.0
bumpversion==0.5.3
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
cloudpickle==1.2.2
colorama==0.4.3
cryptography==2.8
cytoolz==0.10.1
decorator==4.4.1
docopt==0.6.2
docutils==0.16
eth-hash==0.2.0
eth-typing==2.2.1
eth-utils==1.8.4
execnet==1.7.1
flake8==3.4.1
idna==2.8
imagesize==1.2.0
ipython==7.11.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.15.2
jeepney==0.4.2
Jinja2==2.10.3
keyring==21.1.0
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==8.1.0
mypy==0.740
mypy-extensions==0.4.3
packaging==20.0
parso==0.5.2
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.5.0.1
pluggy==0.13.1
prompt-toolkit==3.0.2
ptyprocess==0.6.0
py==1.8.1
pycodestyle==2.3.1
pycparser==2.19
pydocstyle==3.0.0
pyflakes==1.5.0
Pygments==2.5.2
pyparsing==2.4.6
pytest==5.2.2
pytest-asyncio==0.10.0
pytest-forked==1.1.3
pytest-watch==4.2.0
pytest-xdist==1.31.0
pytz==2019.3
PyYAML==5.3
readme-renderer==24.0
requests==2.22.0
requests-toolbelt==0.9.1
SecretStorage==3.1.2
six==1.14.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinx-rtd-theme==0.4.3
sphinxcontrib-websupport==1.1.2
toolz==0.10.0
tox==2.9.1
tqdm==4.41.1
traitlets==4.3.3
twine==3.1.1
typed-ast==1.4.1
typing-extensions==3.7.4.1
urllib3==1.25.8
virtualenv==16.7.9
watchdog==0.9.0
wcwidth==0.1.8
webencodings==0.5.1
Got this failure today on master: https://circleci.com/gh/ethereum/asyncio-run-in-process/374
Retried and it passed. Need to investigate
test_open_proc_SIGINT_can_be_ignored
, test_open_proc_SIGINT_while_running
and test_open_proc_with_trio_KeyboardInterrupt_while_running
all fail randomly when using the open_in_process_with_trio()
runner. That happens because sometimes the runner raises a RuntimeError
like shown below
<function test_open_proc_SIGINT_while_running.<locals>.do_sleep_forever at 0x7f214a0c10d0> raised an unexpected exception: Cancel scope stack corrupted: attempted to exit <trio.CancelScope at 0x7f214a0f6280, active, cancelled> in <Task '__main__._do_async_fn' at 0x7f214a0f28e0> that's still within its child <trio.CancelScope at 0x7f214a0f6460, active, cancelled>
This is probably a bug in your code, that has caused Trio's internal state to
become corrupted. We'll do our best to recover, but from now on there are
no guarantees.
Typically this is caused by one of the following:
- yielding within a generator or async generator that's opened a cancel
scope or nursery (unless the generator is a @contextmanager or
@asynccontextmanager); see https://github.com/python-trio/trio/issues/638
- manually calling __enter__ or __exit__ on a trio.CancelScope, or
__aenter__ or __aexit__ on the object returned by trio.open_nursery();
doing so correctly is difficult and you should use @[async]contextmanager
instead, or maybe [Async]ExitStack
- using [Async]ExitStack to interleave the entries/exits of cancel scopes
and/or nurseries in a way that couldn't be achieved by some nesting of
'with' and 'async with' blocks
- using the low-level coroutine object protocol to execute some parts of
an async function in a different cancel scope/nursery context than
other parts
If you don't believe you're doing any of these things, please file a bug:
https://github.com/python-trio/trio/issues/new
Currently, every process we start is on the same process group as the parent process, and that means a Ctrl-C
in the terminal will cause a SIGINT to be sent to every process we start, but our parent process also has logic to detect a SIGINT and forward it to every child, so by default every child ends up receiving two SIGINTs, unless one adds a start_new_session=True
item to open_in_process()s
subprocess_kwargs` dict, like we do in trinity
As Piper suggested, though, we might want to have that as the default behaviour here
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.