alphatwirl / mantichora

A simple interface to Python multiprocessing and threading

---

A Python library for summarizing event data into multivariate categorical data

Description

AlphaTwirl is a Python library that summarizes event data into multivariate categorical data as data frames. Event data, input to AlphaTwirl, are data with one entry (or row) for one event: for example, data in ROOT TTrees with one entry per collision event of an LHC experiment at CERN. Event data are often large—too large to be loaded in memory—because they have as many entries as events. Multivariate categorical data, the output of AlphaTwirl, have one row for one category. They are usually small—small enough to be loaded in memory—because they only have as many rows as categories. Users can, for example, import them as data frames into R and pandas, which usually load all data in memory, and can perform categorical data analyses with a rich set of data operations available in R and pandas.

---

Quick start

• Jupyter Notebook: Quick start of AlphaTwirl
  

        

---

Publication

• Tai Sakuma, "AlphaTwirl: A Python library for summarizing event data into multivariate categorical data", EPJ Web of Conferences 214, 02001 (2019), doi:10.1051/epjconf/201921402001, 1905.06609

---

License

• AlphaTwirl is licensed under the BSD license.

---

Contact

• Tai Sakuma - [email protected]

mantichora's Issues

implement ThreadingHub.terminate()

mantichora/mantichora/hubthreading.py

Line 154 in 228ab03

def terminate(self):

Debug logging printing to stdout tasks using requests package

Thanks for making this available, it wraps up multiprocessing and threading nicely for my use case!

I'm using this to pull data from a server making requests using the requests package. It works, but the I think the requests.get calls I'm making are printing debug statements for each loop. I'm using atpbar as well and this debug statements are causing the progress bars to be re-printed alternating with the requests debug statements rather than updating.

I tried following the example showing how to use logging and tried that with level=logging.DEBUG, but so far that hasn't prevented the debug statements from being printed. Quite possibly because I've implemented it wrong.

I just tried turning off the logging by doing the following, which worked.

def _configure_logger(self):
    # handler = QueueHandler(self.logging_queue)
    # logger = logging.getLogger()
    # logger.setLevel(logging.DEBUG)
    # logger.addHandler(handler)
    pass

Any suggestions on a better option? I would think the logging example would work if it was re-directing the requests debug statements to a file, so maybe I just have an issue there. Thanks!

stop sorting for receive_finished

This is now inconsistent with README.

• need to stop sorting in MultiprocessingDropbox.poll().

add test for atpbar.flush()

• mantichora/mantichora/hubthreading.py

  Line 142 in 228ab03

  atpbar.flush()

• mantichora/mantichora/hubthreading.py

  Line 178 in 228ab03

  atpbar.flush()

mantichora broken on Windows with release 0.9.8?

First of all I would like to thank you for your work on this awesome package.

With the 0.9.8 release (and more specifically, commit 0216c89), manthicora has stopped working for me on Windows. As far as I understand, fork is not available with multiprocessing on Windows.

Is there a way to possibly revert to previous behaviour on Windows in a future release?

multiprocessing.joinablequeue error when running demo code in jupyter lab

Jupyter Lab: 1.2.6
Python: 3.8.0

Running the following demo code:

Background Jupyter service error logging (and the above cell sinked in running status, cannot be stopped):

Process Worker-2:
Process Worker-1:
Process Worker-3:
Traceback (most recent call last):
Traceback (most recent call last):
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run
self._run_tasks()
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks
message = self.task_queue.get()
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run
self._run_tasks()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks
message = self.task_queue.get()
File "/.pyenv/versions/3.8.0AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>
Traceback (most recent call last):
/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run
self._run_tasks()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks
message = self.task_queue.get()
AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>