libev throws SystemError at the end of execution about goless

[rgalanakis]

Interesting!

• What happens if you git clone the repository and run the tests? We put some flushing between each test to try to make sure this bad behavior happens early rather than late. We had sort of similar things happening in tests at some point but couldn't repro it in real scripts.
• What version of gevent/greenlet/libev?
• Can you send us some repro code, especially so I could try reproing on Linux and under different interpreters? Want to see if its a Windows-only issue with libev, possibly.

[rgalanakis]

So based on some preliminary research, it is likely a Windows-specific thing.

• See if you can repro by running the tests on Windows. I suspect you will be able to.
• If not, see if you can get us a repro to try on Linux.
• I'll see if I can get a Windows VM set up or something to test as well.

[aisola]

I'm thinking the same thing. I tried it on my Ubuntu laptop and it seems to run fine.

Windows 7 Info:

• Python 2.7.6
• gevent 1.0.1
• greenlet 0.4.2 (greenlet installed libev for me).

I REALLY wish I could send you some repro code... However, its a client project and the client is VERY strict about their code... I'll see if I can get the to let me do this (even though it's against contract).

Cloning the repo has the same effect. Odd.

[rgalanakis]

"Cloning the repo has the same effect. Odd." You mean cloning and running the tests under Windows?

[MichaelAz]

I wasn't able to reproduce the weird "throws at the end of running" behavior, but I was able to reproduce the stack trace. This seems to be an issue with gevent, something with running select in the main greenlet.

The program that reproduces the stack trace:

import goless
import greenlet

print "Main running..."
main_greenlet = greenlet.getcurrent()
print "Main greenlet - %s" % main_greenlet
print "Main greenlets parent - %s" % main_greenlet.parent
print

c = goless.chan()

print "Case 1 - recv in main greenlet raises the error"
try:
    c.recv()
except SystemError, e:
    print e
    print

print "Case 2 - select with no greenlets running also raises the error"
try:
    goless.select([goless.rcase(c)])
except SystemError, e:
    print e
    print

print "Case 3 - recv in non-main greenlet works fine"
def recieve():
    print "Case 3, current greenlet - %s" % greenlet.getcurrent()
    print
    for i in range(10):
        c.send("test")
        c.recv()
goless.go(recieve)

print "Case 4 - selecting with greenlet running works fine"
print
goless.select([goless.rcase(c)])

print "Case 5 - to test if this is a goless or gevent issue, this case runs with a raw gevent channel"
import gevent
channel = gevent.queue.Channel()

try:
    channel.get()
    print "If this is printed, it's a goless issue"
    print
except SystemError:
    print "If this is printed, then it's a gevent issue"
    print

print "The program stops running here, no errors should be thrown after this line"

The output for this program:

Main running...
Main greenlet - <greenlet.greenlet object at 0x021D0850>
Main greenlets parent - None

Case 1 - recv in main greenlet raises the error
SystemError: (libev) select: Unknown error
Traceback (most recent call last):
  File "C:\Users\zeev\Desktop\test.py", line 15, in <module>
    c.recv()
  File "C:\Python27\lib\site-packages\goless\channels.py", line 52, in recv
    got = self._recv()
  File "C:\Python27\lib\site-packages\goless\channels.py", line 143, in _recv
    value = self.waiting_chan.receive()
  File "C:\Python27\lib\site-packages\goless\backends.py", line 70, in receive
    return self.get()
  File "C:\Python27\lib\site-packages\gevent\queue.py", line 477, in get
    return waiter.get()
  File "C:\Python27\lib\site-packages\gevent\hub.py", line 569, in get
    return self.hub.switch()
  File "C:\Python27\lib\site-packages\gevent\hub.py", line 332, in switch
    return greenlet.switch(self)
SystemError: (libev) select: Unknown error

Case 2 - select with no greenlets running also raises the error
SystemError: (libev) select: Unknown error
Traceback (most recent call last):
  File "C:\Users\zeev\Desktop\test.py", line 22, in <module>
    goless.select([goless.rcase(c)])
  File "C:\Python27\lib\site-packages\goless\selecting.py", line 71, in select
    _be.yield_()
  File "C:\Python27\lib\site-packages\goless\backends.py", line 86, in yield_
    gevent.sleep()
  File "C:\Python27\lib\site-packages\gevent\hub.py", line 73, in sleep
    waiter.get()
  File "C:\Python27\lib\site-packages\gevent\hub.py", line 569, in get
    return self.hub.switch()
  File "C:\Python27\lib\site-packages\gevent\hub.py", line 332, in switch
    return greenlet.switch(self)
SystemError: (libev) select: Unknown error

Case 3 - recv in non-main greenlet works fine
Case 4 - selecting with greenlet running works fine

Case 3, current greenlet - <Greenlet at 0x2262cb0: safe_wrapped(<function reciev
e at 0x0224DBB0>)>

Case 5 - to test if this is a goless or gevent issue, this case runs with a raw
gevent channel
SystemError: (libev) select: Unknown error
If this is printed, then it's a gevent issue

The program stops running here, no errors should be thrown after this line

[ctismer]

It would be interesting to see if the problem goes away when you switch to using Stackless.
We have put a lot of work into these corner-cases, lately.

regards -- Chris

[rgalanakis]

@MichaelAz running that code under Linux I get a "gevent.hub.LoopExit: This operation would block forever" error in Case 1. So I think the issue is just a bad error message on Windows.

So its just an issue where blocking with no tasklets/greenlets running raises. I can add in a more graceful and consistent way to handle it in goless (wrap the error). I'm sure the behavior in stackless is the same.

[ctismer]

I'm sure the behavior in stackless is the same.

I agree

[rgalanakis]

Ok, the backends will now catch 'deadlock' errors and reraise them as something meaningful. However I need to add the behavior to select as well (check if it is the last running tasklet).
@aisola the current code will not fix your error, but it should make it more clear. I'll polish it tonight.
Strangely though, gevent seems to raise a gevent.hub.LoopExit on Linux, but SystemError on Windows? Can someone with a windows machine run the tests on master's tip?

[aisola]

Tried with Stackless... Same error. I'm no expert, so I'm likely not right, but I believe it's the last tasklet. I think it's just hanging somehow and libev throws some unknown error about it.

[rgalanakis]

Yeah what I think is going on is:

• When doing a channel receive or send on the last tasklet, the underlying backend is raising a RuntimeError in stackless, a gevent.hub.LoopExit error in gevent on Linux, and the uknown SystemError in gevent on Linux. This behavior should now be better handled. A goless.Deadlock exception is raised instead of an implementation-specific error. If you aren't seeing a Deadlock raised with the head of master, there's a bug somewhere.
• When doing a select on the last tasklet, Python will just be in a busy loop, waiting for a channel to be ready. That's what I'm working on now.

Have you tried what's currently in master?

[rgalanakis]

Okay, now select will detect a deadlock. There's still the potential for one, but since there's no fast way to get all running greenlets in gevent (like there is in stackless via stackless.runcount), I erred on the side of performance than correctness. Very easy to move the deadlock check into the select loop if needed or if someone comes up with an optimization. Will create an issue to track this.

[rgalanakis]

@aisola the latest change from @MichaelAz for issue #28 (along with the other work done for deadlock detection) should fix this issue. If this isn't fixed with what's in master, please comment here or open a new ticket.