Coder Social home page Coder Social logo

ethereum / asyncio-run-in-process Goto Github PK

View Code? Open in Web Editor NEW
12.0 12.0 9.0 148 KB

A simple asyncio friendly replacement for multiprocessing to run coroutines in a separate process.

License: MIT License

Shell 2.95% Makefile 1.83% Python 95.22%

asyncio-run-in-process's People

Contributors

carver avatar cburgdorf avatar davesque avatar gsalgado avatar njgheorghita avatar pipermerriam avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

asyncio-run-in-process's Issues

Locally failing tests

What was wrong?

A few tests related to SIGINT and process killing are failing on my machine. I assume this is the cause of ethereum/trinity#1507.

Code that produced the error

The following two test cases are failing:

tests/core/test_open_in_process.py::test_open_proc_SIGINT_can_be_handled FAILED [ 15%]
tests/core/test_open_in_process.py::test_open_proc_SIGINT_can_be_ignored FAILED [ 18%]

Sometimes, they fail immediately, sometimes they block indefinitely and I have to kill them manually.

Full error output

================================================================================== FAILURES ==================================================================================
____________________________________________________________________ test_open_proc_SIGINT_can_be_handled ____________________________________________________________________

    @pytest.mark.asyncio
    async def test_open_proc_SIGINT_can_be_handled():
        async def do_sleep_forever():
            try:
                while True:
                    await asyncio.sleep(0)
            except KeyboardInterrupt:
                return 9999
    
        async with open_in_process(do_sleep_forever) as proc:
            proc.send_signal(signal.SIGINT)
>       assert proc.returncode == 0
E       assert 2 == 0
E         -2
E         +0

do_sleep_forever = <function test_open_proc_SIGINT_can_be_handled.<locals>.do_sleep_forever at 0x7f7cc9b84c10>
proc       = <asyncio_run_in_process.process.Process object at 0x7f7cc9b34430>

tests/core/test_open_in_process.py:58: AssertionError
____________________________________________________________________ test_open_proc_SIGINT_can_be_ignored ____________________________________________________________________

    @pytest.mark.asyncio
    async def test_open_proc_SIGINT_can_be_ignored():
        async def do_sleep_forever():
            try:
                while True:
                    await asyncio.sleep(0)
            except KeyboardInterrupt:
                # silence the first SIGINT
                pass
    
            try:
                while True:
                    await asyncio.sleep(0)
            except KeyboardInterrupt:
                return 9999
    
        async with open_in_process(do_sleep_forever) as proc:
            proc.send_signal(signal.SIGINT)
            await asyncio.sleep(0.01)
>           proc.send_signal(signal.SIGINT)

do_sleep_forever = <function test_open_proc_SIGINT_can_be_ignored.<locals>.do_sleep_forever at 0x7f7cc9b849d0>
proc       = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>

tests/core/test_open_in_process.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>, signum = <Signals.SIGINT: 2>

    def send_signal(self, signum: signal.Signals) -> None:
        """
        Issues the provided signal to the process.
        """
>       os.kill(self.pid, signum.value)
E       ProcessLookupError: [Errno 3] No such process

self       = <asyncio_run_in_process.process.Process object at 0x7f7cc9b4d670>
signum     = <Signals.SIGINT: 2>

asyncio_run_in_process/process.py:277: ProcessLookupError

Environment

$ python -m eth_utils

Python version:
3.8.1 (default, Dec 21 2019, 20:57:38) 
[GCC 9.2.0]

Operating System: Linux-4.19.91-1-MANJARO-x86_64-with-glibc2.2.5

pip freeze result:
alabaster==0.7.12
apipkg==1.5
argh==0.26.2
async-generator==1.10
-e [email protected]:ethereum/asyncio-run-in-process.git@74f11319739c64c0fe8ff7c2aebee3ecf5a125e8#egg=asyncio_run_in_process
atomicwrites==1.3.0
attrs==19.3.0
Babel==2.8.0
backcall==0.1.0
bleach==3.1.0
bumpversion==0.5.3
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
cloudpickle==1.2.2
colorama==0.4.3
cryptography==2.8
cytoolz==0.10.1
decorator==4.4.1
docopt==0.6.2
docutils==0.16
eth-hash==0.2.0
eth-typing==2.2.1
eth-utils==1.8.4
execnet==1.7.1
flake8==3.4.1
idna==2.8
imagesize==1.2.0
ipython==7.11.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.15.2
jeepney==0.4.2
Jinja2==2.10.3
keyring==21.1.0
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==8.1.0
mypy==0.740
mypy-extensions==0.4.3
packaging==20.0
parso==0.5.2
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.5.0.1
pluggy==0.13.1
prompt-toolkit==3.0.2
ptyprocess==0.6.0
py==1.8.1
pycodestyle==2.3.1
pycparser==2.19
pydocstyle==3.0.0
pyflakes==1.5.0
Pygments==2.5.2
pyparsing==2.4.6
pytest==5.2.2
pytest-asyncio==0.10.0
pytest-forked==1.1.3
pytest-watch==4.2.0
pytest-xdist==1.31.0
pytz==2019.3
PyYAML==5.3
readme-renderer==24.0
requests==2.22.0
requests-toolbelt==0.9.1
SecretStorage==3.1.2
six==1.14.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinx-rtd-theme==0.4.3
sphinxcontrib-websupport==1.1.2
toolz==0.10.0
tox==2.9.1
tqdm==4.41.1
traitlets==4.3.3
twine==3.1.1
typed-ast==1.4.1
typing-extensions==3.7.4.1
urllib3==1.25.8
virtualenv==16.7.9
watchdog==0.9.0
wcwidth==0.1.8
webencodings==0.5.1

Some tests fail randomly when using the trio runner

test_open_proc_SIGINT_can_be_ignored, test_open_proc_SIGINT_while_running and test_open_proc_with_trio_KeyboardInterrupt_while_running all fail randomly when using the open_in_process_with_trio() runner. That happens because sometimes the runner raises a RuntimeError like shown below

<function test_open_proc_SIGINT_while_running.<locals>.do_sleep_forever at 0x7f214a0c10d0> raised an unexpected exception: Cancel scope stack corrupted: attempted to exit <trio.CancelScope at 0x7f214a0f6280, active, cancelled> in <Task '__main__._do_async_fn' at 0x7f214a0f28e0> that's still within its child <trio.CancelScope at 0x7f214a0f6460, active, cancelled>

This is probably a bug in your code, that has caused Trio's internal state to
become corrupted. We'll do our best to recover, but from now on there are
no guarantees.

Typically this is caused by one of the following:
  - yielding within a generator or async generator that's opened a cancel
    scope or nursery (unless the generator is a @contextmanager or
    @asynccontextmanager); see https://github.com/python-trio/trio/issues/638
  - manually calling __enter__ or __exit__ on a trio.CancelScope, or
    __aenter__ or __aexit__ on the object returned by trio.open_nursery();
    doing so correctly is difficult and you should use @[async]contextmanager
    instead, or maybe [Async]ExitStack
  - using [Async]ExitStack to interleave the entries/exits of cancel scopes
    and/or nurseries in a way that couldn't be achieved by some nesting of
    'with' and 'async with' blocks
  - using the low-level coroutine object protocol to execute some parts of
    an async function in a different cancel scope/nursery context than
    other parts
If you don't believe you're doing any of these things, please file a bug:
https://github.com/python-trio/trio/issues/new

Consider using a separate session for every child process

Currently, every process we start is on the same process group as the parent process, and that means a Ctrl-C in the terminal will cause a SIGINT to be sent to every process we start, but our parent process also has logic to detect a SIGINT and forward it to every child, so by default every child ends up receiving two SIGINTs, unless one adds a start_new_session=True item to open_in_process()s subprocess_kwargs` dict, like we do in trinity

As Piper suggested, though, we might want to have that as the default behaviour here

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.