Comments (13)
Interesting!
- What happens if you git clone the repository and run the tests? We put some flushing between each test to try to make sure this bad behavior happens early rather than late. We had sort of similar things happening in tests at some point but couldn't repro it in real scripts.
- What version of gevent/greenlet/libev?
- Can you send us some repro code, especially so I could try reproing on Linux and under different interpreters? Want to see if its a Windows-only issue with libev, possibly.
from goless.
So based on some preliminary research, it is likely a Windows-specific thing.
- See if you can repro by running the tests on Windows. I suspect you will be able to.
- If not, see if you can get us a repro to try on Linux.
- I'll see if I can get a Windows VM set up or something to test as well.
from goless.
I'm thinking the same thing. I tried it on my Ubuntu laptop and it seems to run fine.
Windows 7 Info:
- Python 2.7.6
- gevent 1.0.1
- greenlet 0.4.2 (greenlet installed libev for me).
I REALLY wish I could send you some repro code... However, its a client project and the client is VERY strict about their code... I'll see if I can get the to let me do this (even though it's against contract).
Cloning the repo has the same effect. Odd.
from goless.
"Cloning the repo has the same effect. Odd." You mean cloning and running the tests under Windows?
from goless.
I wasn't able to reproduce the weird "throws at the end of running" behavior, but I was able to reproduce the stack trace. This seems to be an issue with gevent, something with running select in the main greenlet.
The program that reproduces the stack trace:
import goless
import greenlet
print "Main running..."
main_greenlet = greenlet.getcurrent()
print "Main greenlet - %s" % main_greenlet
print "Main greenlets parent - %s" % main_greenlet.parent
print
c = goless.chan()
print "Case 1 - recv in main greenlet raises the error"
try:
c.recv()
except SystemError, e:
print e
print
print "Case 2 - select with no greenlets running also raises the error"
try:
goless.select([goless.rcase(c)])
except SystemError, e:
print e
print
print "Case 3 - recv in non-main greenlet works fine"
def recieve():
print "Case 3, current greenlet - %s" % greenlet.getcurrent()
print
for i in range(10):
c.send("test")
c.recv()
goless.go(recieve)
print "Case 4 - selecting with greenlet running works fine"
print
goless.select([goless.rcase(c)])
print "Case 5 - to test if this is a goless or gevent issue, this case runs with a raw gevent channel"
import gevent
channel = gevent.queue.Channel()
try:
channel.get()
print "If this is printed, it's a goless issue"
print
except SystemError:
print "If this is printed, then it's a gevent issue"
print
print "The program stops running here, no errors should be thrown after this line"
The output for this program:
Main running...
Main greenlet - <greenlet.greenlet object at 0x021D0850>
Main greenlets parent - None
Case 1 - recv in main greenlet raises the error
SystemError: (libev) select: Unknown error
Traceback (most recent call last):
File "C:\Users\zeev\Desktop\test.py", line 15, in <module>
c.recv()
File "C:\Python27\lib\site-packages\goless\channels.py", line 52, in recv
got = self._recv()
File "C:\Python27\lib\site-packages\goless\channels.py", line 143, in _recv
value = self.waiting_chan.receive()
File "C:\Python27\lib\site-packages\goless\backends.py", line 70, in receive
return self.get()
File "C:\Python27\lib\site-packages\gevent\queue.py", line 477, in get
return waiter.get()
File "C:\Python27\lib\site-packages\gevent\hub.py", line 569, in get
return self.hub.switch()
File "C:\Python27\lib\site-packages\gevent\hub.py", line 332, in switch
return greenlet.switch(self)
SystemError: (libev) select: Unknown error
Case 2 - select with no greenlets running also raises the error
SystemError: (libev) select: Unknown error
Traceback (most recent call last):
File "C:\Users\zeev\Desktop\test.py", line 22, in <module>
goless.select([goless.rcase(c)])
File "C:\Python27\lib\site-packages\goless\selecting.py", line 71, in select
_be.yield_()
File "C:\Python27\lib\site-packages\goless\backends.py", line 86, in yield_
gevent.sleep()
File "C:\Python27\lib\site-packages\gevent\hub.py", line 73, in sleep
waiter.get()
File "C:\Python27\lib\site-packages\gevent\hub.py", line 569, in get
return self.hub.switch()
File "C:\Python27\lib\site-packages\gevent\hub.py", line 332, in switch
return greenlet.switch(self)
SystemError: (libev) select: Unknown error
Case 3 - recv in non-main greenlet works fine
Case 4 - selecting with greenlet running works fine
Case 3, current greenlet - <Greenlet at 0x2262cb0: safe_wrapped(<function reciev
e at 0x0224DBB0>)>
Case 5 - to test if this is a goless or gevent issue, this case runs with a raw
gevent channel
SystemError: (libev) select: Unknown error
If this is printed, then it's a gevent issue
The program stops running here, no errors should be thrown after this line
from goless.
It would be interesting to see if the problem goes away when you switch to using Stackless.
We have put a lot of work into these corner-cases, lately.
regards -- Chris
from goless.
@MichaelAz running that code under Linux I get a "gevent.hub.LoopExit: This operation would block forever" error in Case 1. So I think the issue is just a bad error message on Windows.
So its just an issue where blocking with no tasklets/greenlets running raises. I can add in a more graceful and consistent way to handle it in goless (wrap the error). I'm sure the behavior in stackless is the same.
from goless.
I'm sure the behavior in stackless is the same.
I agree
from goless.
Ok, the backends will now catch 'deadlock' errors and reraise them as something meaningful. However I need to add the behavior to select as well (check if it is the last running tasklet).
@aisola the current code will not fix your error, but it should make it more clear. I'll polish it tonight.
Strangely though, gevent seems to raise a gevent.hub.LoopExit on Linux, but SystemError on Windows? Can someone with a windows machine run the tests on master's tip?
from goless.
Tried with Stackless... Same error. I'm no expert, so I'm likely not right, but I believe it's the last tasklet. I think it's just hanging somehow and libev throws some unknown error about it.
from goless.
Yeah what I think is going on is:
- When doing a channel receive or send on the last tasklet, the underlying backend is raising a RuntimeError in stackless, a gevent.hub.LoopExit error in gevent on Linux, and the uknown SystemError in gevent on Linux. This behavior should now be better handled. A goless.Deadlock exception is raised instead of an implementation-specific error. If you aren't seeing a Deadlock raised with the head of master, there's a bug somewhere.
- When doing a select on the last tasklet, Python will just be in a busy loop, waiting for a channel to be ready. That's what I'm working on now.
Have you tried what's currently in master?
from goless.
Okay, now select will detect a deadlock. There's still the potential for one, but since there's no fast way to get all running greenlets in gevent (like there is in stackless via stackless.runcount), I erred on the side of performance than correctness. Very easy to move the deadlock check into the select loop if needed or if someone comes up with an optimization. Will create an issue to track this.
from goless.
@aisola the latest change from @MichaelAz for issue #28 (along with the other work done for deadlock detection) should fix this issue. If this isn't fixed with what's in master, please comment here or open a new ticket.
from goless.
Related Issues (20)
- Network-transparent channels? HOT 9
- Find fast way to check for all running greenlets HOT 10
- benchmark.py throws SystemError HOT 13
- PyPyStackless doesn't have a runcount attribute HOT 5
- Install gevent in setup.py HOT 1
- I got an KeyError when use goless. HOT 1
- parallel_for example not using goroutines
- AsyncIO support HOT 8
- ...
- ...
- GO sync package HOT 1
- AttributeError on context switch HOT 4
- Question about goless HOT 3
- Unbuffered channel operations with select at both ends never happen HOT 2
- tickers? HOT 2
- test_raises_deadlock_if_no_goroutines hangs when run with gevent (Python 2.7.13) HOT 5
- SELECT casing is not working HOT 3
- can not parallel when there is "while True" in the routine HOT 5
- channel.close() does not cause an exception on a waiting receiver HOT 1
- Proposing a PR to fix a few small typos
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goless.