Comments (4)
I also get a disturbing error message on shutdown when running a flow in parallel on the command line on Python 3.8.
To reproduce:
- Use Python 3.8.3.
- Check out this branch, which adds some parallelism to the ML workflow example.
- Run the flow with
python -m example.ml_workflow --parallel
. (You can optionally add-C 1e4,1e2,1,1e-2,1e-4
to add more parallelism.)
The flow completes successfully, but I get one of these two error messages afterwards (it's not consistent which one):
Exception in thread Thread-1:
Traceback (most recent call last):
Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x7fd7a54082b0)
Thread 0x000070000b66e000 (most recent call first):
File "/Users/janek/.pyenv/versions/3.8.3/lib/python3.8/threading.py", line 1202 in invoke_excepthook
File "/Users/janek/.pyenv/versions/3.8.3/lib/python3.8/threading.py", line 934 in _bootstrap_inner
File "/Users/janek/.pyenv/versions/3.8.3/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x0000000118dec5c0 (most recent call first):
<no Python frame>
Abort trap: 6
or
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Users/janek/.pyenv/versions/3.8.3/lib/python3.8/threading.py", line 932, in _bootstrap_inner
from bionic.
I don't understand what changed in python 3.8 for this to happen but these exceptions are coming from logging receiver thread at shutdown.
For the shorter exception, I see a similar issue previously reported for python 2.7 but that was supposed to be fixed in 3.2. Maybe this is a regression in 3.8. Either way, the exception seems to be incomplete and it's hard to debug.
For the longer exception, it looks like stderr buffer lock was held to be closed during cleanup and this got stuck. I see an issue where a user was using stdin and had the same exception but our case is different since it's happening in the default except hook (default since we didn't change the except hook for the daemon thread).
I'll keep digging to see if there is an easy to fix to these problems. In the meantime, I'll change the thread name to log receiver thread
so it's a little more obvious in the future.
from bionic.
I finally got the entire exception once.
File "/Users/naman/.pyenv/versions/3.8.3/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/Users/naman/.pyenv/versions/3.8.3/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/Users/naman/Development/bionic/bionic/executor.py", line 170, in _receive
self.queue.empty()
File "<string>", line 2, in empty
File "/Users/naman/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/naman/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/managers.py", line 250, in serve_client
self.id_to_local_proxy_obj[ident]
KeyError: '138d2a4c0'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/naman/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/managers.py", line 252, in serve_client
raise ke
File "/Users/naman/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/managers.py", line 246, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '138d2a4c0'
---------------------------------------------------------------------------```
from bionic.
Fixed in #171
from bionic.
Related Issues (20)
- Errors from unset entities are hard to debug
- @changes_per_run generates extra cache entries
- Non-persistable entities can sometimes be spuriously recomputed
- Redefining an entity with a function can produce surprising results
- The "provider" concept is too broad
- Should bionic.util.init_basic_logging only affect logging for bionic components?
- Compute task states in the subprocesses
- Multiple processes and threads? HOT 1
- Full test suite fails on OS X Catalina HOT 1
- Caching fails when directory names contain spaces
- `FlowBuilder.merge` not accepting another builder is confusing
- Reloading of flow spanning multiple files HOT 1
- Docs: Functions with multiple outputs cannot be called explicitly HOT 2
- Uncached values should be recomputed each time they're used HOT 1
- Set-valued entities are not hashed determistically
- Confusing error message when non-iterable input used with `outputs`
- parallel evaluation inconsistencies with persist settings
- Improve error message when protocols fail
- Test vector update script should delete old test vectors
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bionic.