alphatwirl / mantichora Goto Github PK

Thanks for making this available, it wraps up multiprocessing and threading nicely for my use case!

I'm using this to pull data from a server making requests using the requests package. It works, but the I think the requests.get calls I'm making are printing debug statements for each loop. I'm using atpbar as well and this debug statements are causing the progress bars to be re-printed alternating with the requests debug statements rather than updating.

I tried following the example showing how to use logging and tried that with level=logging.DEBUG, but so far that hasn't prevented the debug statements from being printed. Quite possibly because I've implemented it wrong.

I just tried turning off the logging by doing the following, which worked.

def _configure_logger(self):
    # handler = QueueHandler(self.logging_queue)
    # logger = logging.getLogger()
    # logger.setLevel(logging.DEBUG)
    # logger.addHandler(handler)
    pass

Any suggestions on a better option? I would think the logging example would work if it was re-directing the requests debug statements to a file, so maybe I just have an issue there. Thanks!

Jupyter Lab: 1.2.6
Python: 3.8.0

Running the following demo code:

Background Jupyter service error logging (and the above cell sinked in running status, cannot be stopped):

Process Worker-2:
Process Worker-1:
Process Worker-3:
Traceback (most recent call last):
Traceback (most recent call last):
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run
self._run_tasks()
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks
message = self.task_queue.get()
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run
self._run_tasks()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks
message = self.task_queue.get()
File "/.pyenv/versions/3.8.0AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>
Traceback (most recent call last):
/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 235, in run
self._run_tasks()
File "/.pyenv/versions/3.8.0/lib/python3.8/site-packages/mantichora/hubmp.py", line 250, in _run_tasks
message = self.task_queue.get()
AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>
File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'task_loop' on <module 'main' (built-in)>

This is now inconsistent with README.

• need to stop sorting in MultiprocessingDropbox.poll().

First of all I would like to thank you for your work on this awesome package.

With the 0.9.8 release (and more specifically, commit 0216c89), manthicora has stopped working for me on Windows. As far as I understand, fork is not available with multiprocessing on Windows.

Is there a way to possibly revert to previous behaviour on Windows in a future release?

• mantichora/mantichora/hubthreading.py

  Line 142 in 228ab03

  atpbar.flush()

• mantichora/mantichora/hubthreading.py

  Line 178 in 228ab03

  atpbar.flush()