cherrypy / magicbus

The Process Bus is a publish/subscribe architecture that loosely connects components with services.

Welcome to the GitHub repository of CherryPy!

CherryPy is a pythonic, object-oriented HTTP framework.

1. It allows building web applications in much the same way one would build any other object-oriented program.
2. This design results in more concise and readable code developed faster. It's all just properties and methods.
3. It is now more than ten years old and has proven fast and very stable.
4. It is being used in production by many sites, from the simplest to the most demanding.
5. And perhaps most importantly, it is fun to work with :-)

Here's how easy it is to write "Hello World" in CherryPy:

import cherrypy

class HelloWorld(object):
    @cherrypy.expose
    def index(self):
        return "Hello World!"

cherrypy.quickstart(HelloWorld())

And it continues to work that intuitively when systems grow, allowing for the Python object model to be dynamically presented as a website and/or API.

While CherryPy is one of the easiest and most intuitive frameworks out there, the prerequisite for understanding the CherryPy documentation is that you have a general understanding of Python and web development. Additionally:

• Tutorials are included in the repository: https://github.com/cherrypy/cherrypy/tree/master/cherrypy/tutorial
• A general wiki at: https://github.com/cherrypy/cherrypy/wiki

If the docs are insufficient to address your needs, the CherryPy community has several avenues for support.

For Enterprise

CherryPy is available as part of the Tidelift Subscription.

The CherryPy maintainers and the maintainers of thousands of other packages are working with Tidelift to deliver one enterprise subscription that covers all of the open source you use.

Learn more.

Contributing

Please follow the contribution guidelines. And by all means, absorb the Zen of CherryPy.

magicbus's Issues

Signals tests are unreliable

Tests under magicbus/test/test_signals.py spawn processes and send signals to them. Then they use magicbus.plugins.opsys.PIDFile plugin to wait for subprocess to reload. Its join() method waits for the PID file to stop existing and then exits.

The problem is that when receiving certain signals, subprocesses "reload", not exit permanently. And that reloading flow recreates the PID. This works in some envs that are rather slow (namely Travis CI) but on my laptop, it blocks on join() forever because the subprocess reloads and creates a new PID file faster than the join() makes another iteration of its loop. So join() "thinks" that this file keeps existing which in fact is not true because it's a new file now.
And I think this is what's happening in AppVeyor CI too: https://ci.appveyor.com/project/CherryPy/magicbus/builds/31405204/job/61xrbpxhyurlx0w0#L78.

So this test suite works incidentally. It must be made more robust.

This also means that magicbus.plugins.opsys.PIDFile.join() implementation is buggy and has to be refactored.

Intermittent failures in test_start_with_callback

As observed in this job, some test runs will fail with

======================================================================
FAIL: test_start_with_callback (magicbus.test.test_processbus.BusMethodTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/cherrypy/magicbus/magicbus/test/test_processbus.py", line 273, in test_start_with_callback
    self.assertEqual(events, ['g', ('f', (1, 3, 5), {'foo': 'bar'})])
AssertionError: Lists differ: [('f', (1, 3, 5), {'foo': 'bar... != ['g', ('f', (1, 3, 5), {'foo':...

First differing element 0:
('f', (1, 3, 5), {'foo': 'bar'})
'g'

- [('f', (1, 3, 5), {'foo': 'bar'}), 'g']
?                                  -----

+ ['g', ('f', (1, 3, 5), {'foo': 'bar'})]
?  +++++
----------------------------------------------------------------------

Other times, the test passes just fine. I observed the same behavior on macOS. Likely there's some structure that's dependent on dict or set ordering.

`Bus.wait()` raises `ValueError`

>>> import os
>>> import magicbus

>>> while os.open("/dev/null", 0) < 1023: pass
>>> bus = magicbus.Bus()
>>> bus.wait('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../magicbus/magicbus/base.py", line 352, in wait
    _wait()
  File ".../magicbus/magicbus/base.py", line 325, in _wait
    r, w, x = select.select([read_fd], [], [], interval)
ValueError: filedescriptor out of range in select()

Looks like a limitation of select.select

I already made PR #20 to try to fix it

PyPI publishing setup for GHA

@jaraco since I don't have access to this package on PyPI, could you please add a PYPI_TOKEN secret in this repo settings?
Or just add me to the package maintainers on PyPI and then I can do it myself. The GHA setup that needs it is #13.

Fix licensing

This project has ambiguous license (BSD 3-clause in LICENSE, MIT in Trove Classifiers).

How to fix? CherryPy seems to have BSD 3-clause, so I'll update the classifier to match.

`test_processbus.py::TestBusMethod::test_wait` flakily fails with `START_ERROR` because of `EINVAL`

Huh?

The test in question is unstable and it's mostly visible in Windows runs but I think I also saw similar Ubuntu failures.

Example test failure: https://github.com/cherrypy/magicbus/runs/2114870045#step:12:106

The log appears to be:

Bus state: ENTER
Bus state: IDLE
Bus state: START
Bus state: START_ERROR
Exiting due to error in start listener:
Traceback (most recent call last):
  File "D:\a\magicbus\magicbus\magicbus\base.py", line 205, in _transition
    os.write(write_fd, b'1')
OSError: [Errno 22] Invalid argument
Bus state: STOP

Pointers

magicbus.base.Bus.wait() creates a pipe for getting the transition notifications more efficiently and adds it to the instance attribute _state_transition_pipes: https://github.com/cherrypy/magicbus/blob/57d151e/magicbus/base.py#L315-L334
This only happens if the runtime is select-compatible.
It then enters a loop waiting for either the select.select() to get a notification about any transition change or just sleeps for a while otherwise.

magicbus.base.Bus.transition() (through magicbus.base.Bus._transition()) writes a byte into every pipe in _state_transition_pipes attempting to notify the waiters before their timeouts expire: https://github.com/cherrypy/magicbus/blob/57d151e/magicbus/base.py#L202-L205

Performing the transitions in separate threads may create a race condition when list(self._state_transition_pipes) got a copy of the pipes, then during the iteration, some pipe file descriptors get closed and removed by wait() asynchronously between that copy and the loop iteration attempting to write into the previously copied descriptors.
This probably means that the hack of avoiding the "Set changed size during iteration" is not clever enough and we need to suppress the error that happens there.
One thing that is surprising to me is that I cannot reproduce this behavior under GNU/Linux on my machine — it always returns other errors (like "Broken pipe" if the reader end of the pipe has been closed before writing or "Bad file descriptor" if the write fd has been closed). Does Windows produce different errors?

Another thing to consider is that when the fd is closed, a slightly different race may still be possible — when a new pipe has been opened and the kernel allocated exactly the same file descriptor that the old discarded pipe had but is still in the variable. It's not that bad if we just hit another pipe created by wait() but may be dangerous if that file descriptor is pointing at something else completely. We should probably introduce some locking mechanism to prevent this.

5d4cfe2 should help us see the future exceptions since they weren't logged properly before.

Refs I encountered when I was trying to explore how write() could cause EINVAL:

• torvalds/linux@9883035
• https://man7.org/linux/man-pages/man2/pipe.2.html
• https://stackoverflow.com/a/28020096/595220
• https://bugs.python.org/issue19612
• https://hg.python.org/cpython/rev/039ac3f01c4e
• https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer
• https://man7.org/linux/man-pages/man3/errno.3.html
• https://stackoverflow.com/a/6031694/595220
• https://man7.org/linux/man-pages/man2/write.2.html
• https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/write?view=msvc-160
• https://github.com/python/cpython/blob/148bc05/Modules/posixmodule.c#L10004-L10084
• https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe
• https://stackoverflow.com/questions/23688492/oserror-errno-22-invalid-argument-in-subprocess
• https://hg.python.org/cpython/file/62cf4e77f785/Lib/subprocess.py#l1199
• https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/pipe?view=msvc-160
• https://github.com/python/cpython/blob/148bc05/Modules/clinic/_winapi.c.h#L346-L361
• gevent/gevent#148
• 116229c (#160)
• https://bugs.python.org/issue15896

Weird stuff

I was trying to reproduce this specific type of OSError on my Gentoo machine under IPython by using os.pipe() and trying to break the file handles but never succeeded to trick it into returning EINVAL (OSError: [Errno 22] Invalid argument). I haven't tried adding threads, though.

@aminusfu do you have any insight into this?

Related archeology:

• os.pipe() was first introduced by @aminusfu in #1
• That change wasn't inspected deeply during the review: https://github.com/cherrypy/magicbus/pull/1/files#r597829271
• The follow-up refactoring may also have a number of https://github.com/cherrypy/magicbus/pull/4/files#r597832583