doodspav / atomics Goto Github PK
View Code? Open in Web Editor NEWLock-Free Atomics in Python
License: GNU General Public License v3.0
atomics's People
Contributors
Stargazers
Watchers
Forkers
museghostatomics's Issues
Required package for Android development
Hi there.
Could you provide a package for Android development ?
In the development with Beeware "https://docs.beeware.org/en/latest/tutorial/tutorial-7.html" some packages must be available in the Chaquopy package repository "https://chaquo.com/pypi-7.0/".
So it will require it for ARM processors.
Can it be done ?
Greetings
UnsupportedWidthException on long byte array
Hi, I am using your library to atomically store data in a shared memory buffer. This works fine when I store just a few bytes (for example I have a struct with 2 integers and 2 floats). However, I recently wanted to store a struct with 100 characters along with 10 integers. I received the error below. I cannot find much information on the error in the code or the documentation. Am I doing something wrong or is this a limitation of the library? Are there any workarounds I could implement?
File "venv\Lib\site-packages\atomics\_impl\atomic\funcs.py", line 109, in atomicview
return AtomicBytesViewContext(buffer=buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "venv\Lib\site-packages\atomics\_impl\atomic\bytes.py", line 22, in __init__
super().__init__(buffer=buffer, is_integral=False, is_signed=False)
File "venv\Lib\site-packages\atomics\_impl\atomic\base.py", line 79, in __init__
raise UnsupportedWidthException(width, readonly=ro)
atomics._impl.exceptions.UnsupportedWidthException: No operations are supported on objects with a width of 110.
Error when running with Python -OO optimizations
I am getting the following stack trace and exception when running code with Python -OO:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/ronny/Projects/py/UltraDict/./examples/parallel.py:37 in <module> │
│ │
│ 34 │ │
│ 35 │ # No name provided to create a new dict with random name. │
│ 36 │ # To make it work under Windows, we need to set a static `full_dump_size` │
│ ❱ 37 │ ultra = UltraDict(buffer_size=10_000, shared_lock=True, full_dump_size=10_000) │
│ 38 │ ultra['counter'] = 0 │
│ 39 │ │
│ 40 │ # Our children will use the name to attach to the existing dict │
│ │
│ /home/ronny/Projects/py/UltraDict/../UltraDict/UltraDict.py:453 in __init__ │
│ │
│ 450 │ │ # Local lock for all processes and threads created by the same interpreter │
│ 451 │ │ if shared_lock: │
│ 452 │ │ │ try: │
│ ❱ 453 │ │ │ │ self.lock = self.SharedLock(self, 'lock_remote', 'lock_pid_remote') │
│ 454 │ │ │ except NameError: │
│ 455 │ │ │ │ #self.cleanup() │
│ 456 │ │ │ │ raise Exceptions.MissingDependency("Install `atomics` Python package to │
│ │
│ /home/ronny/Projects/py/UltraDict/../UltraDict/UltraDict.py:109 in __init__ │
│ │
│ 106 │ │ │ self.init_pid() │
│ 107 │ │ │ │
│ 108 │ │ │ try: │
│ ❱ 109 │ │ │ │ self.ctx = atomics.atomicview(buffer=self.lock_remote[0:1], atype=atomic │
│ 110 │ │ │ │ self.pid_remote_ctx = atomics.atomicview(buffer=self.pid_remote[0:4], at │
│ 111 │ │ │ except NameError as e: │
│ 112 │ │ │ │ self.cleanup() │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/atomics/_impl/atomic/funcs.py:109 in atomicview │
│ │
│ 106 │ elif atype is UINT: │
│ 107 │ │ return AtomicUintViewContext(buffer=buffer) │
│ 108 │ elif atype is BYTES: │
│ ❱ 109 │ │ return AtomicBytesViewContext(buffer=buffer) │
│ 110 │ elif atype is INTEGRAL: │
│ 111 │ │ return AtomicIntegralViewContext(buffer=buffer, **kwargs) │
│ 112 │ elif atype is ANY: │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/atomics/_impl/atomic/bytes.py:22 in __init__ │
│ │
│ 19 class AtomicBytesViewContext(AtomicViewContext): │
│ 20 │ │
│ 21 │ def __init__(self, *, buffer): │
│ ❱ 22 │ │ super().__init__(buffer=buffer, is_integral=False, is_signed=False) │
│ 23 │ │
│ 24 │ def __enter__(self) -> AtomicBytesView: │
│ 25 │ │ self._assert_enter_preconditions() │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/atomics/_impl/atomic/base.py:65 in __init__ │
│ │
│ 62 │ │ pybuf = None │
│ 63 │ │ try: │
│ 64 │ │ │ with memoryview(buffer) as view: │
│ ❱ 65 │ │ │ │ pybuf = PyBuffer(buffer, writeable=(not view.readonly)) │
│ 66 │ │ except TypeError: │
│ 67 │ │ │ pass │
│ 68 │ │ # check for TypeError; raise outside exception handler for nicer error message │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/atomics/_impl/pybuffer.py:18 in __init__ │
│ │
│ 15 │ │ │ if writeable and view.readonly and not force: │
│ 16 │ │ │ │ raise RuntimeError("Cannot create writeable PyBuffer from readonly expor │
│ 17 │ │ │ # get and save buffer │
│ ❱ 18 │ │ │ self._buf = ffi.from_buffer("char[]", exporter, not view.readonly) │
│ 19 │ │ │ self._obj = exporter │
│ 20 │ │ │ self._len = view.nbytes │
│ 21 │ │ │ self._readonly = not writeable │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/api.py:364 in from_buffer │
│ │
│ 361 │ │ if python_buffer is _unspecified: │
│ 362 │ │ │ cdecl, python_buffer = self.BCharA, cdecl │
│ 363 │ │ elif isinstance(cdecl, basestring): │
│ ❱ 364 │ │ │ cdecl = self._typeof(cdecl) │
│ 365 │ │ return self._backend.from_buffer(cdecl, python_buffer, │
│ 366 │ │ │ │ │ │ │ │ │ │ require_writable) │
│ 367 │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/api.py:186 in _typeof │
│ │
│ 183 │ │ │ result = self._parsed_types[cdecl] │
│ 184 │ │ except KeyError: │
│ 185 │ │ │ with self._lock: │
│ ❱ 186 │ │ │ │ result = self._typeof_locked(cdecl) │
│ 187 │ │ # │
│ 188 │ │ btype, really_a_function_type = result │
│ 189 │ │ if really_a_function_type and not consider_function_as_funcptr: │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/api.py:171 in _typeof_locked │
│ │
│ 168 │ │ if not isinstance(cdecl, str): # unicode, on Python 2 │
│ 169 │ │ │ cdecl = cdecl.encode('ascii') │
│ 170 │ │ # │
│ ❱ 171 │ │ type = self._parser.parse_type(cdecl) │
│ 172 │ │ really_a_function_type = type.is_raw_function │
│ 173 │ │ if really_a_function_type: │
│ 174 │ │ │ type = type.as_function_pointer() │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/cparser.py:552 in parse_type │
│ │
│ 549 │ │ │ │ │ │ self._declare('variable ' + decl.name, tp, quals=quals) │
│ 550 │ │
│ 551 │ def parse_type(self, cdecl): │
│ ❱ 552 │ │ return self.parse_type_and_quals(cdecl)[0] │
│ 553 │ │
│ 554 │ def parse_type_and_quals(self, cdecl): │
│ 555 │ │ ast, macros = self._parse('void __dummy(\n%s\n);' % cdecl)[:2] │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/cparser.py:555 in parse_type_and_quals │
│ │
│ 552 │ │ return self.parse_type_and_quals(cdecl)[0] │
│ 553 │ │
│ 554 │ def parse_type_and_quals(self, cdecl): │
│ ❱ 555 │ │ ast, macros = self._parse('void __dummy(\n%s\n);' % cdecl)[:2] │
│ 556 │ │ assert not macros │
│ 557 │ │ exprnode = ast.ext[-1].type.args.params[0] │
│ 558 │ │ if isinstance(exprnode, pycparser.c_ast.ID): │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/cparser.py:336 in _parse │
│ │
│ 333 │ │ if lock is not None: │
│ 334 │ │ │ lock.acquire() # pycparser is not thread-safe... │
│ 335 │ │ try: │
│ ❱ 336 │ │ │ ast = _get_parser().parse(fullcsource) │
│ 337 │ │ except pycparser.c_parser.ParseError as e: │
│ 338 │ │ │ self.convert_pycparser_error(e, csource) │
│ 339 │ │ finally: │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/cffi/cparser.py:53 in _get_parser │
│ │
│ 50 def _get_parser(): │
│ 51 │ global _parser_cache │
│ 52 │ if _parser_cache is None: │
│ ❱ 53 │ │ _parser_cache = pycparser.CParser() │
│ 54 │ return _parser_cache │
│ 55 │
│ 56 def _workaround_for_old_pycparser(csource): │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/pycparser/c_parser.py:109 in __init__ │
│ │
│ 106 │ │ for rule in rules_with_opt: │
│ 107 │ │ │ self._create_opt_rule(rule) │
│ 108 │ │ │
│ ❱ 109 │ │ self.cparser = yacc.yacc( │
│ 110 │ │ │ module=self, │
│ 111 │ │ │ start='translation_unit_or_empty', │
│ 112 │ │ │ debug=yacc_debug, │
│ │
│ /home/ronny/.local/lib/python3.9/site-packages/pycparser/ply/yacc.py:3352 in yacc │
│ │
│ 3349 │ │ errors = True │
│ 3350 │ │
│ 3351 │ if errors: │
│ ❱ 3352 │ │ raise YaccError('Unable to build parser') │
│ 3353 │ │
│ 3354 │ # Verify the grammar structure │
│ 3355 │ undefined_symbols = grammar.undefined_symbols() │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
This is also in line with -OO which removed the doc string but it seems this is necessary for CFFI?
could not find patomic
$ pip install -U atomics
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Requirement already satisfied: atomics in d:\program\miniconda\lib\site-packages (1.0.2)
Requirement already satisfied: cffi>=1.10 in d:\program\miniconda\lib\site-packages (from atomics) (1.15.1)
Requirement already satisfied: pycparser in d:\program\miniconda\lib\site-packages (from cffi>=1.10->atomics) (2.21)
Performance rather slow
First of all many thanks for your great library. It's the only one I've found that can do atomic operations on a shared memory in Python. I'm using it for locking in my dictionary implementation that uses shared memory.
Though, I was wondering what kind of performance can be expected? On the Github README you have an incorrect example with a counter up to 10 million and a correct one that uses the atomics library (but the counter does only go to 10k.
The incorrect one takes 0.34 s on my computer and the correct one using atomics library takes 41.38 s if I count to the same target of 10 million.
If I correct the incorrect one in a classical way using threading.RLock() it takes 3.68 s which is still factor 10 faster.
In my dictionary implementation, I'm using AtomicBytesView.exchange() to set my shared lock because it seemed faster to compare myself than using cmpxchg_weak(). I've also found that load() seems rather slow.
Do you have any idea how to get better (similar) performance to RLock() when using atomics for setting a lock flag in shared memory? This is basically setting a byte to 1 or 0.
Add arm64 for Mac M1 to the release distribution
I'm working on an M1 and unfortunately the build wheel for arm64 is not provided. It would be awesome to provide it to a new release on pypi so that Mac M1 is natively supported.
I got around it by building it from source for now, I assume other people who would use atomics on M1 would have the same problem.
I can offer my help for this.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.