osvenskan / posix_ipc Goto Github PK

View Code? Open in Web Editor NEW

129.0 129.0 22.0 827 KB

License: Other

Python 52.48% C 47.52%

posix_ipc's People

Contributors

Stargazers

Watchers

posix_ipc's Issues

Using setuptools breaks source distribution build

Creating a post-1.0.0 release is stalled due to setuptools supporting fewer options than Python's distutils.

Background

Python's distutils allows using a MANIFEST.in files to specify the files you want to include in a source distribution. As part of the distribution build process, Python translates MANIFEST.in to MANIFEST. Python's distutils also allows you to create a MANIFEST file directly, skipping the MANIFEST.in intermediate step. This is the way I've been doing things.

The Problem

setuptools doesn't support reading MANIFEST directly; it only looks for MANIFEST.in. Because this project doesn't have a MANIFEST.in, setuptools ignores my MANIFESET and uses default settings to decide which files should go in the tarball. This creates a tarball with hardly anything in it. It misses the demo files and, critically, the prober on which setup.py relies.

I added setuptools to setup.py after 1.0.0 was released in order to create wheels for 1.0.0. This is the first release I've tried to make using setup.py with setuptools in it, which is why this is the first time this problem has occurred.

Travis CI doesn't support Python 3.x under OS X

I'd like to add support for testing under OS X but Travis doesn't support Python under that OS yet.

Convert doc to Markdown

Summary

Currently my personal site is the primary repository for the documentation. Since it's formatted in HTML, it's less easy to read on GitHub than it could be. If the doc was in Markdown, it's be easier to change the project's organization to have GitHub be the primary source of both the doc and source code.

SharedMemory constructor w/size should not resize existing segment

Reported by Thomas T -
on opening a shared memory segment with O_CREAT and a size, the memory segment will be resized with ftruncate if the size is not zero (line 1040). I think this behavior is a bug, since the shared memory might already be existing and be larger, for example. I suggest that you add some way to make it an error if open succeeds but the size does not match, or to somehow disallow resizing of the shared memory segment.

Comment from me -
This behavior contradicts my own documentation for the SharedMemory class constructor which says, "With flags set to O_CREAT, the module opens the segment if it exists (in which case size and mode are ignored)”. I don’t want to change the behavior (existing code might rely on it), so my initial reaction is to change the doc to instruct users to pass 0 for the size if they don’t want to take a chance on accidentally resizing an existing segment.

Missing hash for v1.1.1

Using poetry (and I suspect pipenv is the same) I can't install version 1.1.1, it's failing with;

Package operations: 1 install, 0 updates, 0 removals

  • Installing posix-ipc (1.1.1): Failed

  RuntimeError

  Hash for posix-ipc (1.1.1) from archive posix_ipc-1.1.1-cp310-cp310-linux_x86_64.whl not found in known
hashes  (was: sha256:5894b2133418e9d388ecbef76b2e1b3cc8e06e506bb56635aebc0ab125d0d0a7)

Not entirely sure what the build / deployment mechanism is for wheels, but it works fine for 1.1.0, so I'm guessing something isn't quite right somewhere. For anyone with problems;

poetry add [email protected]

Works fine. Installing with "pip" also works fine, which makes me think the package is there but something has screwed up the hashes. Last time I saw this, someone had managed to re-publish using the same version number ... but it could also be something a little less desirable (!)

Context manager cleanup

On Linux, this script leaves an orphaned semaphore showing with ls -l /dev/shm. If the semaphore was created by the context manager, it seems like it would be safe to unlink it when exiting the context manager scope. Is that something that can be added to the library?

For now, I'm working around it by wrapping everything in a finally clause that performs the cleanup.

#!/usr/bin/env python

import posix_ipc

with posix_ipc.Semaphore(name='myname', flags=posix_ipc.O_CREAT, initial_value=1):
    print(1)

Change default values for message queue

Hi there. Thanks for the amazing lib.

Could you, please, provide some guidance as to how I can change the default values for the message queue?

I have used the command: sudo sysctl fs.mqueue for each file in /proc/sys/fs/mqueue/
My actual system reads:
fs.mqueue.msg_default = 360
fs.mqueue.msg_max = 500
fs.mqueue.msgsize_default = 921646
fs.mqueue.msgsize_max = 1382500
fs.mqueue.queues_max = 25

At first, when I tried to create a new message queue posix_ipc wouldn't show these values. It kept on showing the previous default values.
Reading through the code I saw that you run the prober.py once at installation time. So I understood that I needed to reinstall posix_ipc.
When I did that posix_ipc read the new values, but when I try to create the message queue it shows this error:

Exception has occurred: OSError
This process already has the maximum number of files open

Is there anything else I need to do?
Thanks in advance.

pip install within a container fails.

I wanted to use this in a docker container but it seems it cannot be installed in a docker image.
Here is my Dockerfile

FROM alpine:3.18

RUN apk --no-cache update
RUN apk --no-cache add libbpf libelf libgcc \
        libstdc++ clang16-libs llvm16-libs \
        musl xz-libs python3
RUN apk add py3-pip

RUN pip3 install posix_ipc

CMD ["sleep","12324"]

and here is the errors i get wen i build the image

------                                                                                                                                                                                       
 > [5/6] RUN pip3 install posix_ipc:                                                                                                                                                         
1.488 Collecting posix_ipc                                                                                                                                                                   
1.578   Downloading posix_ipc-1.1.1.tar.gz (94 kB)                                                                                                                                           
1.616      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.3/94.3 kB 2.6 MB/s eta 0:00:00                                                                                                        
1.658   Installing build dependencies: started                                                                                                                                               
5.394   Installing build dependencies: finished with status 'done'
5.396   Getting requirements to build wheel: started
6.238   Getting requirements to build wheel: finished with status 'done'
6.250   Preparing metadata (pyproject.toml): started
6.653   Preparing metadata (pyproject.toml): finished with status 'done'
6.667 Building wheels for collected packages: posix_ipc
6.669   Building wheel for posix_ipc (pyproject.toml): started
6.875   Building wheel for posix_ipc (pyproject.toml): finished with status 'error'
6.889   error: subprocess-exited-with-error
6.889   
6.889   × Building wheel for posix_ipc (pyproject.toml) did not run successfully.
6.889   │ exit code: 1
6.889   ╰─> [22 lines of output]
6.889       ******************************************************************************
6.889       * Setup can't determine if it needs to link to the realtime libraries on your
6.889       * system, so it will default to 'no' which may not be correct.
6.889       *
6.889       * Please report this message and your operating system info to the package
6.889       * maintainer listed in the README file.
6.889       ******************************************************************************
6.889       ******************************************************************************
6.889       * Setup can't determine the value of PAGE_SIZE on your system, so it will
6.889       * default to 4096 which may not be correct.
6.889       *
6.889       * Please report this message and your operating system info to the package
6.889       * maintainer listed in the README file.
6.889       ******************************************************************************
6.889       running bdist_wheel
6.889       running build
6.889       running build_ext
6.889       building 'posix_ipc' extension
6.889       creating build
6.889       creating build/temp.linux-x86_64-cpython-311
6.889       gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include/python3.11 -c posix_ipc_module.c -o build/temp.linux-x86_64-cpython-311/posix_ipc_module.o
6.889       error: command 'gcc' failed: No such file or directory
6.889       [end of output]
6.889   
6.889   note: This error originates from a subprocess, and is likely not a problem with pip.
6.889   ERROR: Failed building wheel for posix_ipc
6.890 Failed to build posix_ipc
6.890 ERROR: Could not build wheels for posix_ipc, which is required to install pyproject.toml-based projects
------
Dockerfile:9
--------------------
   7 |     RUN apk add py3-pip
   8 |     
   9 | >>> RUN pip3 install posix_ipc
  10 |     COPY transmitter.py transmitter.py
  11 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install posix_ipc" did not complete successfully: exit code: 1

Any ideas how to fix this??

Does posix_ipc's message queue thread safe & process safe?

For example, In the case that I create a single MessageQueue() instance and multiple processes(by multiprocessing module) share that instance, does multiple processes calling MessageQueue.send() at the same time cause race condition?
I know that posix message queue itself is thread safe, but I'm not sure this wrapper of python is also safe.

Travis CI doesn't support Python 3.7 yet

I'd like to add support for testing under Python 3.7 but Travis doesn't support it yet.

MessageQueue receive doesn't take timeout parameter

It looks like the documentation for MessageQueue.receive is incorrect. It shows the signature as receive([timeout = None]), but if I try to pass a timeout I get TypeError: function takes at most 1 argument (2 given).

It looks like the source indeed doesn't receive a timeout parameter. Was this deliberate?

Thanks a ton for this library!

Version discrepancy between pypi and github

Could you please either release 1.0.4 & 1.0.5 on pypi and/or tag pypi's apparent "1.1.0" in this repo? The discrepancy between the two sets off alarm bells in the heads of cautious people over whether the pypi account has been hijacked etc.

Universal wheel for linux

Dear Mr Semanchuk

I would like to kindly ask you if you could release a universal python3 wheel package for linux.

Honestly, I'm not very experienced with github or travis ci, so I can't competently deliver you a pull request with the necessary automation.

I used to release your package as wheel into our company's PyPi repository by myself, but as a security measure our IT has changed our PyPI mirror so that we cannot upload packages without the required prefix.

I firmly believe that the Linux wheel package will be a benefit for everyone.

Regards

Jozef

Cross compiling for Arm

Please provide some simple instructions for cross compiling posix_ipc whrein:
Compilation host is X86 Linux and
Intended target is Arm 32 bit

Methods `fileno()` for compatiability with `selectors`

The selectors module can register objects which export a fileno() method which returns the file descriptor integer.

This is already exported by, for example, mqd and fd, but maybe a method for fileno() could also be added?

A current workaround is to extend the classes from posix_ipc and add the method.

Support for M1 Macs?

When trying to use this module on macOS on an ARM machine, this test fails on the following line:

posix_ipc/tests/test_memory.py

Line 315 in 88c384a

self.assertEqual(mem.size, new_size)

With this message: AssertionError:16384 != 4096

This is more than likely due to these processors using a 16k page size rather than the more usual 4k. The Asahi Linux community has already written much about this.

SharedMemory should permit file descriptor to be 0

Reported by Thomas T --
A valid file descriptor can be zero. If you close file descriptor 0 (usually stdin), then you will get 0 on the next open call. As far as I can see, your code assumes that the descriptor is positive or negative, and zero is an invalid file descriptor (lines 938 and 994 in posix_ipc_module.c). If you get descriptor 0 then `SharedMemory_close_fd' cannot close it (line 1127).

Error Compiling manylinux2010 Wheel

Hi and thank you so much for this library! It's been helping scientists perform rapid IO and apply multi-processing to large 3D images.

I've been trying to generate binary wheels for one of my libraries, but posix_ipc has been generating an error during compilation. This is odd, because installing from source on a target machine works. It's only during wheel generation that this happens.

The first error that the compiler spits out is:

    posix_ipc_module.c:1567:37: error: invalid conversion from ‘void*’ to ‘const char*’ [-fpermissive]
             rc = mq_send(self->mqd, msg.buf, msg.len, (unsigned int)priority);

It seems that this is something I might be able to fix by providing appropriate compiler flags, but it seems like this could result in undefined behavior. Is there something I can safely do about this?

Here's the commands that were run:

    Running setup.py install for posix-ipc: started
    Running setup.py install for posix-ipc: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /opt/_internal/cpython-3.7.9/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1g9dkw8w/posix-ipc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1g9dkw8w/posix-ipc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8yzle7l9/install-record.txt --single-version-externally-managed --compile --install-headers /opt/_internal/cpython-3.7.9/include/python3.7m/posix-ipc
         cwd: /tmp/pip-install-1g9dkw8w/posix-ipc/
    Complete output (81 lines):
    running install
    running build
    running build_ext
    building 'posix_ipc' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/_internal/cpython-3.7.9/include/python3.7m -c posix_ipc_module.c -o build/temp.linux-x86_64-3.7/posix_ipc_module.o

Oddly, if I log into the base docker image docker run -it -v $PWD:/posix_ipc quay.io/pypa/manylinux2010_x86_64 /bin/bash and try to run /opt/python/cp37-cp37m/bin/python3.7 setup.py develop it compiles normally.

It's crashing during this docker script:

FROM quay.io/pypa/manylinux2010_x86_64 
ENV CC "g++"
RUN /opt/python/cp37-cp37m/bin/pip3.7 install posix-ipc

Semaphore class gives "Invalid Parameter" error on Ubuntu

HI:
I'm trying to use the Semaphore class like this:

sem = posix_ipc.Semaphore(name= <some string>, flags=posix_ipc.O_CREAT, initial_value=1)

However, it keeps telling me

ValueError: Invalid parameter(s)

I looked at the examples given on this repo, but really cannot identify the problem.
I would really appreciate your opionins. Thank you!

`Semaphore::release` / `::acquire` segfault on Linux

The library causes a bus error when calling acquire or release on a semaphore. Message queues work fine on the same machine.

On a Raspberry Pi 4 w/ Ubuntu:

$ pip show posix_ipc
Name: posix-ipc
Version: 1.1.1
Summary: POSIX IPC primitives (semaphores, shared memory and message queues) for Python
Home-page: https://github.com/osvenskan/posix_ipc
Author: Philip Semanchuk
Author-email: [email protected]
License: http://creativecommons.org/licenses/BSD/
Location: /usr/local/lib/python3.10/dist-packages
Requires: 
Required-by: 

$ touch /dev/shm/sem.foobar
$ python3
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import posix_ipc
>>> sem = posix_ipc.Semaphore('/foobar')
>>> sem.release()
Bus error (core dumped)

Same version of the library on an x86 machine w/ Debian, with a more recent Python version from Homebrew:

$ touch /dev/shm/sem.foobar
$ python3         
Python 3.11.5 (main, Aug 24 2023, 12:23:19) [GCC 13.1.0] on linux     
Type "help", "copyright", "credits" or "license" for more information.
>>> import posix_ipc                                                  
>>> sem = posix_ipc.Semaphore('/foobar')                              
>>> sem.release()                                                     
Bus error

'gcc' failed: No such file or directory: 'gcc'

Hi there,

I'm trying to install posix_ipc on my linux machine (Ubuntu 20.04.6 LTS; python 3.7.16; in conda venv) but am having some difficulty. I've cloned the repository and tried installing w/ setup.py and get the following:

...~/Documents/posix_ipc$ python setup.py install
******************************************************************************
* Setup can't determine if it needs to link to the realtime libraries on your
* system, so it will default to 'no' which may not be correct.
* 
* Please report this message and your operating system info to the package
* maintainer listed in the README file.
******************************************************************************
******************************************************************************
* Setup can't determine the value of PAGE_SIZE on your system, so it will
* default to 4096 which may not be correct.
* 
* Please report this message and your operating system info to the package
* maintainer listed in the README file.
******************************************************************************
running install
/home/marcel/anaconda3/envs/synapse/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  setuptools.SetuptoolsDeprecationWarning,
/home/marcel/anaconda3/envs/synapse/lib/python3.7/site-packages/setuptools/command/easy_install.py:147: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  EasyInstallDeprecationWarning,
running bdist_egg
running egg_info
writing posix_ipc.egg-info/PKG-INFO
writing dependency_links to posix_ipc.egg-info/dependency_links.txt
writing top-level names to posix_ipc.egg-info/top_level.txt
reading manifest file 'posix_ipc.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.txt' under directory 'demo3'
warning: no files found matching '*.txt' under directory 'demo4'
warning: no files found matching '*.txt' under directory 'demo5'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'posix_ipc.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'posix_ipc' extension
gcc -pthread -B /home/marcel/anaconda3/envs/synapse/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/marcel/anaconda3/envs/synapse/include/python3.7m -c posix_ipc_module.c -o build/temp.linux-x86_64-cpython-37/posix_ipc_module.o
error: command 'gcc' failed: No such file or directory: 'gcc'

I've also tried pip install posix_ipc, but this also doesn't work:

~/Documents/posix_ipc$ pip install posix_ipc
Collecting posix_ipc
  Using cached posix_ipc-1.1.1.tar.gz (94 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: posix_ipc
  Building wheel for posix_ipc (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      ******************************************************************************
      * Setup can't determine if it needs to link to the realtime libraries on your
      * system, so it will default to 'no' which may not be correct.
      *
      * Please report this message and your operating system info to the package
      * maintainer listed in the README file.
      ******************************************************************************
      ******************************************************************************
      * Setup can't determine the value of PAGE_SIZE on your system, so it will
      * default to 4096 which may not be correct.
      *
      * Please report this message and your operating system info to the package
      * maintainer listed in the README file.
      ******************************************************************************
      running bdist_wheel
      running build
      running build_ext
      building 'posix_ipc' extension
      creating build
      creating build/temp.linux-x86_64-cpython-37
      gcc -pthread -B /home/marcel/anaconda3/envs/synapse/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/marcel/anaconda3/envs/synapse/include/python3.7m -c posix_ipc_module.c -o build/temp.linux-x86_64-cpython-37/posix_ipc_module.o
      error: command 'gcc' failed: No such file or directory: 'gcc'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for posix_ipc
  Running setup.py clean for posix_ipc
Failed to build posix_ipc
Installing collected packages: posix_ipc
  Running setup.py install for posix_ipc ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for posix_ipc did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      ******************************************************************************
      * Setup can't determine if it needs to link to the realtime libraries on your
      * system, so it will default to 'no' which may not be correct.
      *
      * Please report this message and your operating system info to the package
      * maintainer listed in the README file.
      ******************************************************************************
      ******************************************************************************
      * Setup can't determine the value of PAGE_SIZE on your system, so it will
      * default to 4096 which may not be correct.
      *
      * Please report this message and your operating system info to the package
      * maintainer listed in the README file.
      ******************************************************************************
      running install
      /home/marcel/anaconda3/envs/synapse/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        setuptools.SetuptoolsDeprecationWarning,
      running build
      running build_ext
      building 'posix_ipc' extension
      creating build
      creating build/temp.linux-x86_64-cpython-37
      gcc -pthread -B /home/marcel/anaconda3/envs/synapse/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/marcel/anaconda3/envs/synapse/include/python3.7m -c posix_ipc_module.c -o build/temp.linux-x86_64-cpython-37/posix_ipc_module.o
      error: command 'gcc' failed: No such file or directory: 'gcc'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> posix_ipc

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Thanks for any advice!

message queue raise exception when receives a signal

Hi there,

When I use a signal inside my program, the posix_ipc.MessageQueue object raises a posix_ipc.SignalError exception; I believe this is different from what we normally expect, example program to show this problem:

import os
import signal

import posix_ipc


def receive_handler(signal, stack):
    print(f"Received signal: {signal}")


q = posix_ipc.MessageQueue("/test", flags=posix_ipc.O_CREAT)

signal.signal(signal.SIGUSR1, receive_handler)

print(f'Runing (PID: {os.getpid()}) ...')
q.receive()

After running this program, if you send a signal with another process, let's say with kill command:

 kill -s USR1 5426

You get:

Running (PID: 5426) ...
Received signal: 10
Traceback (most recent call last):
  File "/home/bahram/.virtualenvs/tmp-1237b73b1e1465c/main.py", line 16, in <module>
    q.receive()
posix_ipc.SignalError: The wait was interrupted by a signal

Comparing this to other queues like queue.Queue they catch the signal and after doing the sigaction continue their functionality.

Problem of messageQueue on macos

Hello, I would like to know why I don't have a messageQueue on macos. However, if I use this library on Ubuntu, then messageQueue can be used. Is this a problem on my side or it just the lib doesn't support messageQueue on macOS? I have already tried all possible solutions and searched for information of this problem in many sources, but I did not find the answer. I hope that you will give me an answer to my question, thanks!

distutils deprecated

Per PEP 632, distutils will raise a deprecation warning as of Python 3.10 and will be totally gone as of Python 3.12. posix_ipc's setup.py uses distutils so that needs to be fixed.

https://www.python.org/dev/peps/pep-0632/

ftruncate() should be called judiciously under OS X

Reported by Tomas Z -
Apple's implementation of ftruncate only works once and when called on a new memory block. Second and later calls on the same block (whether be created or just opened for writing) always fail with EINVAL (even when shrinking the block) and that makes the module very hard to use on macOS, as it is constantly failing telling the user that "The size is invalid or the memory is read-only".

DPRINTF fails under Python 3

Reported by Tomas Z -
When debugging is enabled by means of the POSIX_IPC_DEBUG conditional, the module cannot be compiled under Python 3.x since PyString_Check doesn't seem to exist.

OSError when executing test_message_queues.py unittest

When I run the test_message_queues.py tests I get the error OSError: This process already has the maximum number of files open for various cases. I am executing it on the commit with the tag rel1.0.5. Why is this happening?

The output looks as follows:

 ❯ python tests/test_message_queues.py
...EE..E.E.E.E.......E......................
======================================================================
ERROR: test_name_as_bytes (__main__.TestMessageQueueCreation)
Test that the name can be bytes.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 135, in test_name_as_bytes
    mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_name_as_unicode (__main__.TestMessageQueueCreation)
Test that the name can be unicode.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 154, in test_name_as_unicode
    mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_o_creat_new (__main__.TestMessageQueueCreation)
tests posix_ipc.O_CREAT to create a new MessageQueue without
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 105, in test_o_creat_new
    mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREAT)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_randomly_generated_name (__main__.TestMessageQueueCreation)
tests that the randomly-generated name works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 118, in test_randomly_generated_name
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_read_flag_new_queue (__main__.TestMessageQueueCreation)
test that the read flag is respected on a new queue
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 179, in test_read_flag_new_queue
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX, read=False)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_write_flag_new_queue (__main__.TestMessageQueueCreation)
test that the write flag is respected on a new queue
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 194, in test_write_flag_new_queue
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX, write=False)
OSError: This process already has the maximum number of files open

======================================================================
ERROR: test_block_flag_default_value_and_writability (__main__.TestMessageQueuePropertiesAndAttributes)
test that the block flag is True by default and can be changed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/raka/Workspace/idp/posix_ipc/tests/test_message_queues.py", line 548, in test_block_flag_default_value_and_writability
    mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX)
OSError: This process already has the maximum number of files open

----------------------------------------------------------------------
Ran 44 tests in 2.006s

FAILED (errors=7)

PyEval_ThreadsInitialized and PyEval_InitThreads are deprecated

Hi,
Building the module with Python 3.9 shows that some deprecated functions are called:

posix_ipc_module.c: In function ‘MessageQueue_request_notification’:
posix_ipc_module.c:1923:9: warning: ‘PyEval_ThreadsInitialized’ is deprecated [-Wdeprecated-declarations]
 1923 |         if (!PyEval_ThreadsInitialized()) {
      |         ^~
In file included from /usr/include/python3.9/Python.h:145,
                 from posix_ipc_module.c:33:
/usr/include/python3.9/ceval.h:129:36: note: declared here
  129 | Py_DEPRECATED(3.9) PyAPI_FUNC(int) PyEval_ThreadsInitialized(void);
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~
posix_ipc_module.c:1925:13: warning: ‘PyEval_InitThreads’ is deprecated [-Wdeprecated-declarations]
 1925 |             PyEval_InitThreads();
      |             ^~~~~~~~~~~~~~~~~~
In file included from /usr/include/python3.9/Python.h:145,
                 from posix_ipc_module.c:33:
/usr/include/python3.9/ceval.h:130:37: note: declared here
  130 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~

Import Error, undefined symbol: shm_unlink

I cloned a project from github and installed it on Ubuntu, but it cant' run. And it arises a ERROR as below which is related to posix_ipc.

File "/homeb/tangwuguo/projects/raptorX/DL4DistancePrediction4/DataProcessor.py", line 11, in <module>
    from shared_ndarray import SharedNDArray
  File "build/bdist.linux-x86_64/egg/shared_ndarray/__init__.py", line 3, in <module>
    
ImportError: /homeb/tangwuguo/miniconda3/envs/raptorX/lib/python2.7/site-packages/posix_ipc.so: undefined symbol: shm_unlink

pip install error with version 1.1.0 on Docker

#11 4.562   Downloading https://files.pythonhosted.org/packages/41/72/90b6702782921c0e3e0b10a66f2a47d8cfaf9c2255d763ab45cc8cea25eb/posix_ipc-1.1.0.tar.gz (94kB)
#11 4.686     Complete output from command python setup.py egg_info:
#11 4.686     Traceback (most recent call last):
#11 4.686       File "<string>", line 1, in <module>
#11 4.686       File "/tmp/pip-build-wwawuefb/posix-ipc/setup.py", line 20, in <module>
#11 4.686         long_description = f.read().strip()
#11 4.686       File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
#11 4.686         return codecs.ascii_decode(input, self.errors)[0]
#11 4.686     UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 300: ordinal not in range(128)
#11 4.686
#11 4.686     ----------------------------------------
#11 4.694 Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-wwawuefb/posix-ipc/

missing 1.0.5 tag?

I'm not seeing the 1.0.5 tag and corresponding release in github. Cheers.

SEM_FAILED now defined correctly on Mac OS

Per this comment (https://github.com/python/cpython/pull/11664/files#r253295800), Mac OS is now distributing a header file with a correct definition of SEM_FAILED, so the redefinition dance in posix_ipc_module.c can be removed.

shared memory object leaked.

I pressed CTRL + C during demo/premise.py running, but now I can't start it again:

(tensorflow_python3) chenjiasheng@ubuntu:~/code/posix_ipc/demo$ python premise.py
[email protected]: Oooo 'ello, I'm Mrs. Premise!
Traceback (most recent call last):
File "premise.py", line 21, in
size=params["SHM_SIZE"])
posix_ipc.ExistentialError: Shared memory with the specified name already exists

Removing /run/shm/four_yorkshiremen dosen't solve it.

(tensorflow_python3) chenjiasheng@ubuntu:~/code/posix_ipc/demo$ ll /run/shm
total 4
drwxrwxrwt 2 root root 80 Jan 17 10:14 ./
drwxr-xr-x 33 root root 1200 Jan 16 17:12 ../
-rw------- 1 chenjiasheng chenjiasheng 4096 Jan 17 10:14 four_yorkshiremen
-rw------- 1 chenjiasheng chenjiasheng 32 Jan 17 09:41 sem.wuthering_heights

Re-open and unlink/close_fd the shared memory object dosen't work either.
I don't like to reboot or change the memory object name.
Please help me out. Thank you!

Problems in the execution of setup.py

Hi, I'm trying tu use your code on a Windows machine, when I execute the code in Visual Studio he says:
Traceback (most recent call last): File "c:/Users/Lorenzo/Desktop/posix_ipc-master/setup.py", line 13, in <module> VERSION = open("VERSION").read().strip() FileNotFoundError: [Errno 2] No such file or directory: 'VERSION' .
I dont't know what can I do to fix the problem.

segfault w/msg queue threaded rearm under BSD

The test test_request_notification_threaded_rearm() sometimes segfaults on FreeBSD. I don't know why it should segfault, but running with DPRINTF enabled is informative. Under Linux, the DPRINTF log shows an orderly, repeated sequence of events --

MessageQueue_request_notification is called by the main thread (once)
PyEval_InitThreads is called once
MessageQueue_request_notification exits
mq_send() is called
A new C thread is invoked
The C thread invokes the Python callback
The callback requests notification, which results in a 'Notification cancelled' entry in the log
MessageQueue_request_notification() exits
'Done calling' marks the end of the callback
Sometime after this point the thread exits
mq_receive() is called (by the test case)
mq_send() is called (by the test case) and the cycle repeats

Implicit in the description above is the behavior of the Threading.Event object which the test case relies on to awake the main thread when the callback is invoked. Once the main thread is woken, it should quickly call mq.receive(). (Keep in mind that notifications should only be sent when the queue transitions from empty to not-empty.)

Under FreeBSD, the sequence of events is diferent. Everything appears normal until after the first call to mq_receive() which is immediately followed by the first call to mq_send(). After that, the log shows a loop of over 500,000 iterations of callback/rearm without mq_receive() being called.

I'm suspicious of the code in uipc_mqueue.c marked by this comment --

/*
 * if there is no receivers and message queue
 * is not empty, we should send notification
 * as soon as possible.
 */

Improve debug mode

Right now, debug mode (enabling DPRINTF statements) requires a recompile to enable/disable. It'd be nicer if that was available all the time but switched off by default. Ideas for turning on debug statements at runtime include --

Different classes (e.g. SemaphoreDebug)
Flag passed at init (e.g. sem = Semaphore(..., debug=True))
Context manager (e.g. with posix_ipc.DebugMode:)
Global on/off flag (e.g. posix_ipc.debug = True)

I also want to consider proper logging instead of always writing to stdout.

Suggestions are welcome. :-)

osvenskan / posix_ipc Goto Github PK

posix_ipc's People

Contributors

Stargazers

Watchers

Forkers

posix_ipc's Issues

Background

The Problem

Summary

Recommend Projects

Recommend Topics

Recommend Org