The installation doesn't work as mentioned in INSTALL. If I install using python setup.py install, none of the cython modules gets installed properly. Eg. import coro would fail (saying no module _coro). Maybe something to do with the setup file. Same applies to aplib. I could get around this by python setup.py build_ext --inplace.

There's a lot of code with handwritten 'def' wrappers for 'cdef' funs & methods. Many/most of these could probably be collapsed into single cpdefs.

5: Thu Jan 31 03:06:40 2013 thread 5 (server connection on ('10.0.2.2', 59957)): error '('http/server.py push|224', "<class 'zlib.error'>", 'Error -2 while flushing: inconsistent stream state', '[_coro.pyx coro._coro._wrap1 (coro/_coro.c:10093)|751] [http/server.py run|98] [http/server.py error|305] [http/server.py done|235] [http/server.py push|224]')'

I'm writing unit tests for my shrapnel application code and I'd like to use the unit test running code in the coro_unittest module. The problem is that it's in the "test" directory which is not included when shrapnel is installed. If that module were moved to the "coro" package (possibly in a "test" sub-package) then it would be available to the unit tests for the shrapnel client applications. My workaround for now is to copy coro_unittest.py into my application code, but it would be nice if I didn't have to do that.

Trigger a yield from <main> (say, with a socket recv()), and you'll segfault.
In _coro.pyx:1289:

cdef _YIELD():
return (the_scheduler._current).__yield ()

_current is None when in <main>.

I'm guessing that the kqueue signal capture is failing for some reason?
If I call the sigterm handler manually it exits correctly.
It's possible that kevent() is not seeing the event, because when I attach with gdb after hitting ^C it is sitting in kevent doing nothing.

Did we forget to include coro's xmlrpc lib or should I port python's xmlrpc ?

My hack that checks for llvm doesn't appear to work with pre-Lion gcc.
That gcc reports itself as:

gcc --version
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)

But requires the 16-byte offset, not the 8-byte offset.
Perhaps it's not llvm that distinguishes the different offset, but the platform?

This leads to the immediate segfault. Anyone have a pre-Lion machine to test this on?

If a coroutine blocks for a long period of time, and then exits, a latency warning is not displayed. I was a little surprised by this.

Expecting something like this:
2015-04-11T01:25:35.991801 'boot status' 0 255 '00000000'

Getting this a lot:
2015-04-11T01:29:58.355978 'boot status' 0 255 ' \x01\xcf '

Single-byte tags are limited to the range 0-30. When used in a protocol implementation with the APPLICATION flag set, this limits the number of different packet types that can be flagged in the protocol. While 31 types should be enough for most protocols, it precludes the ability to use a 'private' range of tags (e.g. tags > 100 indicate private protocol extensions).

Unfortunately the assumption that tag & flags fit into a single byte was pretty heavily built in to the codec, so fixing it required sweeping changes. I've placed this work into the multibyte-tag branch.

But I'd actually prefer to not use that code. I have a different code base that uses a nifty trick for encoding that will give much higher performance (by avoiding lots of copying). This different code base also has extensive auto-generated tests.

The current asn.1 codec maps some constructs directly to their corresponding objects, (INTEGER, SEQUENCE, etc...) and maps everything else to a tuple of (kind, tag, data).

It might be nice to use derived classes to indicate exact asn.1 types. This would also help to make the codec a round-trip one: you can feed the output of the decoder to the encoder and get back the original asn.1.

For example, have a BITSTRING class derived from bytes, have an ENUM class derived from int, etc...

With these changes we could still have efficient and easy-to-use output.

I've done a ton of work today on 1) fixing the pxd files in coro and 2) making it possible to implement servers in pure cython. See my repository "shrapnel-pure-cython-server" for a sample. All the work has been done in the 'pdxfix' branch. Anyone wants to take a look at it before I merge it back to master, let me know.

I've recently learned how to use Sphinx to (mostly) autogenerate documentation, and then publish the docs on github.
I'd like to start doing the same with Shrapnel... There are already lots of doc strings, don't know if they will need translation from whatever form they are in or if they'll just work.
Just wanted to get feedback from y'all on this idea. Here's what the output looks like:
http://samrushing.github.com/amqp-shrapnel/

I saw a python.core today that came from packet.pyx. I couldn't get gdb to load the symbols properly but from the traceback it seems that it received an empty response and we proceeded to unpack it. The daemon has been running for a month or so now and that's the first time I see this issue.

https://github.com/ironport/shrapnel/blob/master/coro/dns/stub_resolver.py#L43

I added a check to make sure that the reply is not empty before we pass it to unpacker. I'll see if it happens again.

I can provide the python core if anyone is interested in it.

@samrushing I'm sure we can catch this and give a better error message.

$ catlog -a 10 /var/log/output.log
Traceback (most recent call last):
  File "/usr/local/bin/catlog", line 4, in <module>
    __import__('pkg_resources').run_script('coro==1.0.5', 'catlog')
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 723, in run_script
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1643, in run_script
  File "/usr/local/lib/python2.7/dist-packages/coro-1.0.5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/catlog", line 110, in <module>

  File "/usr/local/lib/python2.7/dist-packages/coro-1.0.5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/catlog", line 40, in main

  File "/usr/local/lib/python2.7/dist-packages/coro-1.0.5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/catlog", line 67, in seek

AttributeError: 'NoneType' object has no attribute 'groups'

The dependency on aplib causes a lot of complexity with installation. It would be best to remove the dependency, and the drawback of copying some stuff into shrapnel.

The _coro.coro.TimeoutError class inherits from _coro.coro.Interrupted, which, in turn, inherits from BaseException. Simple Exception is not subclassed. Therefore, timeouts cannot be trapped so:

    try:
        coro.with_timeout(timeout, method)
    except Exception, unused:
        handle_exception()

Instead, one must use:

    try:
        coro.with_timeout(timeout, method)
    except (Exception, TimeoutError), unused:
        handle_exception()

According to documentation, BaseException is not to be directly inherited by user-defined classes. Likewise, BaseException is used in terminal situations which are not meant to be (normally) intercepted by the user. For example, GeneratorExit is a subclass of BaseException, while StopIteration subclasses Exception. Users are expected to handle StopIteration, while GeneratorExit is not often observed in userland.

Generally, I think timeout situation should not be considered as a terminal condition. If there is a possibility to impose an explicit timeout on a some operation, I think it should be possible to trap and handle the related exception on the same level, as anticipated, and not terminal, situation.

There's no way to make a connection to a link local address, because the 'address tuple' is an ipv4-style 2-tuple rather than an ipv6-style 4-tuple. See the ssh_ipv6 branch for a fix. Needs more testing from non-ssh ipv6 users.

This may be similar to issue #39. Any exceptions raised during asynchronous operation can't be trapped using try/except. Using set_exception_notifier(...) works though.

I think this is because client.read_thread() method fails to capture and handle exceptions when calling client._read_message(). Further, request and latch objects miss any exception handling functionality.

The following hack worked for me:

from coro.http.client import client as http_client

class http_kludge(http_client):

    def read_thread(self):
        while 1:
            req = self.pending.pop()
            if req is None:
                break
            else:
                try:
                    self._read_message(req)
                except Exception, exc:
                    req.latch.cv.raise_all(exc) # put the exception where it belongs
                else:
                    if not req.response:
                        break
                    else:
                        req.wake()

Modern sshd installs seem to disable the older 'password' authentication by defaut?
Regardless, implementing this should allow for stuff like 2FA.
Somebody should do it.

event_queue.pyx does a little wrapping of an STL priority_queue, this can probably be done directly in Cython now.
[in other words, coro/event_queue.cc and coro/event_queue.h can probably go away]

All the version strings refer to perforce revision strings that no longer apply.

There are bunch of errors on lio.pyx. Also ssh/auth/userauth.py uses a dict comprehension which doesn't work on python 2.6

The Packer class lacks the offset attribute, which it would use to track the resource data length of RRs:

1: Tue May 1 12:17:36 2012 thread 1 (<function dns_recv at 0x1081b7320>): error '('packet.pyx coro.dns.packet.Packer.addRRheader (coro/dns/packet.c:5860)|284', "<type 'exceptions.AttributeError'>", "'coro.dns.packet.Packer' object has no attribute 'offset'", '[_coro.pyx coro._coro._wrap1 (coro/_coro.c:9806)|770] [radardns.py dns_recv|39] [radardns.py respond_with_cname|50] [packet.pyx coro.dns.packet.Packer.addCNAME (coro/dns/packet.c:6354)|290] [packet.pyx coro.dns.packet.Packer.addCNAME (coro/dns/packet.c:6218)|291] [packet.pyx coro.dns.packet.Packer.addRRheader (coro/dns/packet.c:5860)|284]')'

Using TSC has a variety of problems.

• Determining which sysctl to read to determine the TSC-to-seconds conversion is difficult. Some systems don't even have it (My Mac Mini with Snow Leopard, Bob's Amazon FreeBSD Xen virtualized system, etc.).
• Some systems do not have a reliable TSC. We do not check for this.
• There's clunky code to handle re-calibrating our relationship of TSC to wall clock. I've never been comfortable with this. (This was a performance hack to avoid calling gettimeofday a lot, I'm not sure how important that is.)

We need a new API for handling high-resolution time. Preferably something that can use multiple types of time. See ironport/aplib#2 for more detail.

Preferably, this new API would be in shrapnel to help remove the dependency with aplib.

Hello Guys,

I found that sslip module (wrapper for the OpenSSL library) is missing in the shrapnel code tree. Because of this fact the coroutine/coro_ssl.py module is broken and I cannot use SSL connections with shrapnel. If the module was forgotten when shrapnel went open source please add it. If it was decided not to post the sslip here for some reason please let me know.

Thanks.

I'm aware that the modernize merge broke linux, I'm fixing it right now.

To Repro: run any of the websocket demos. The websocket implementation uses a with_timeout wrapper around a read_packet() call, in order to send a 'pong' over the connection if nothing has been seen in ten seconds.

This works fine on freebsd/osx, but fails with a bizarre SimultaneousError on linux when the timeout expires.

10: Tue Nov 19 21:35:45 2013 thread 10 (<bound method terminal.read_thread of <__main__.terminal instance at 0x10cc098>>): error '(\'linux_poller.pyx coro._coro.queue_poller.set_wait_for (coro/_coro.c:17363)|207\', "<class \'coro._coro.SimultaneousError\'>", "<SimultaneousError co=<coro #10 name=\'<bound method terminal.read_thread of <__main__.terminal instance at 0x10cc098>>\' dead=0 started=1 scheduled=0 at 0x107bbc0> other=<coro #10 name=\'<bound method terminal.read_thread of <__main__.terminal instance at 0x10cc098>>\' dead=0 started=1 scheduled=0 at 0x107bbc0> event=<event_key events=1 fd=6>>", \'[_coro.pyx coro._coro._wrap1 (coro/_coro.c:10482)|751] [http/websocket.py read_thread|151] [_coro.pyx coro._coro.sched.with_timeout (coro/_coro.c:14199)|1094] [http/websocket.py read_packet|165] [coro/read_stream.py read_exact|93] [coro/read_stream.py <genexpr>|91] [coro/read_stream.py gen_read_exact|73] [coro/read_stream.py next|8] [ssl/__init__.py recv|91] [ssl/__init__.py _non_blocking_retry|53] [socket.pyx coro._coro.sock.wait_for_read (coro/_coro.c:25492)|925] [socket.pyx coro._coro.sock._wait_for_read (coro/_coro.c:25590)|939] [linux_poller.pyx coro._coro.queue_poller._wait_for_read (coro/_coro.c:18369)|287] [linux_poller.pyx coro._coro.queue_poller._wait_for_with_eof (coro/_coro.c:18288)|280] [linux_poller.pyx coro._coro.queue_poller._wait_for (coro/_coro.c:18482)|295] [linux_poller.pyx coro._coro.queue_poller.set_wait_for (coro/_coro.c:17363)|207]\')'

A temporary workaround for websockets on linux is to remove the with_timeout() call completely.

This is pretty simple, just need to add aliases and/or wrappers for enter() and exit() to support the 'with' keyword.

A query like select sum(value) from test will return a string value rather than an integer value.

The read_stream module didn't make it in, so probably several things in the coroutine directory are broken.
Specifically, I noticed that the httpd server won't work without it.

If there's mostly only one thread running (say, a tight network loop), the stack is getting evacuated and restored for no reason. Instead, wait to evacuate the stack until another thread wants to run there instead. When resuming, if stack_copy is NULL, then _restore() is a no-op. [I thought we already did this, but on examining the code it seems not]

I think we should just remove all the version stuff. Nothing supports CVS tags.

def with_timeout(coro.sock conn):
try:
    coro.with_timeout(io_method, conn)
except coro.TimeoutError, unused:
    cleanup()
...

Inside the cleanup method, I cannot do anything meaningful without getting "notify_of_close ... unable to interrupt thread" message (this includes deferring cleanup to another coroutine).

Interestingly, if I don't trap coro.TimeoutError, coro.event_loop exits with coro.DeadCoroutine exception.

What is a correct way to impose timeout on IO operation? On a coroutine?

I've got a failing build on an Archlinux 64-bit box with Python3.3 installed from source:

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
            :Exceptions:
                - `ValueError`: The value could not be parsed.
            """
            cdef sockaddr_in * sin
            cdef sockaddr_in6 *sin6
            cdef sockaddr_un * sun
                ^
    ------------------------------------------------------------

    coro/socket.pyx:990:13: 'sockaddr_un' is not a type identifier

I've traced the problem to the Cython compiler comparing UNAME_SYSNAME against "Linux" with one encoded as bytes and the other as a unicode string in socket.pxd:

    cdef extern from "sys/un.h":
        IF UNAME_SYSNAME == "Linux":  # I'm broken
            cdef struct sockaddr_un:
                short sun_family
                char sun_path[104]
        ELSE:
            pass

I'm not sure if this is a bug here or upstream. I think the official Cython string philosophy may require a -3 flag to the Cython build in my use-case.

I tried using stub_resolver yesterday and I noticed that when you do stub_resolver.install(), the_resolver doesn't change until reload(coro).

This is a python behavior but I am wondering if we can workaround it in a better way.

from coro.dns import stub_resolver
stub_resolver.install(['8.8.8.8'])
import coro
coro.the_resolver
<coro._coro.dummy_resolver object at 0x8015590b0>
reload(coro)
coro.the_resolver
<coro.dns.stub_resolver.stub_resolver instance at 0x801a1ed40>

The probe for libaio works for finding the library, but does not probe for the includes.

Sam: email an SSH key to [email protected] if you'd like an account to test in my environment.

log:

syadasti@charlie:$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.3 LTS"
syadasti@charlie:$ uname -a
Linux charlie 3.5.0-40-generic #62precise1-Ubuntu SMP Fri Aug 23 17:38:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
syadasti@charlie:$ python --version
Python 2.7.3
syadasti@charlie:$ cython --version
Cython version 0.15.1
syadasti@charlie:$ git clone https://github.com/ironport/shrapnel.git
Cloning into 'shrapnel'...
remote: Counting objects: 2083, done.
remote: Compressing objects: 100% (876/876), done.
remote: Total 2083 (delta 1214), reused 2032 (delta 1175)
Receiving objects: 100% (2083/2083), 1.01 MiB | 294 KiB/s, done.
Resolving deltas: 100% (1214/1214), done.
syadasti@charlie:$ cd shrapnel
syadasti@charlie:/shrapnel$ python setup.py build
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.26.tar.gz
Extracting in /tmp/tmpsDqnRw
Now working in /tmp/tmpsDqnRw/distribute-0.6.26
Building a Distribute egg in /home/syadasti/shrapnel
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
/home/syadasti/shrapnel/distribute-0.6.26-py2.7.egg
test/build/test_lio.c:8:23: fatal error: sys/event.h: No such file or directory
compilation terminated.
detected NPN-capable OpenSSL
/usr/lib/python2.7/distutils/extension.py:133: UserWarning: Unknown Extension options: 'cython_compile_time_env'
warnings.warn(msg)
/usr/lib/python2.7/distutils/extension.py:133: UserWarning: Unknown Extension options: 'pyrex_compile_time_env'
warnings.warn(msg)
running build
running build_py
file backdoor.py (for module backdoor) not found
file coro_process.py (for module coro_process) not found
file coro_unittest.py (for module coro_unittest) not found
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/coro
copying coro/init.py -> build/lib.linux-x86_64-2.7/coro
copying coro/read_stream.py -> build/lib.linux-x86_64-2.7/coro
copying coro/backdoor.py -> build/lib.linux-x86_64-2.7/coro
copying coro/print_profile.py -> build/lib.linux-x86_64-2.7/coro
copying coro/optional.py -> build/lib.linux-x86_64-2.7/coro
copying coro/frontdoor.py -> build/lib.linux-x86_64-2.7/coro
copying coro/signal_handler.py -> build/lib.linux-x86_64-2.7/coro
copying coro/profiler.py -> build/lib.linux-x86_64-2.7/coro
copying coro/tb.py -> build/lib.linux-x86_64-2.7/coro
creating build/lib.linux-x86_64-2.7/coro/http
copying coro/http/websocket.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/handlers.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/http_date.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/client.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/spdy.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/json_rpc.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/session_handler.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/server.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/init.py -> build/lib.linux-x86_64-2.7/coro/http
copying coro/http/protocol.py -> build/lib.linux-x86_64-2.7/coro/http
creating build/lib.linux-x86_64-2.7/coro/ssl
copying coro/ssl/init.py -> build/lib.linux-x86_64-2.7/coro/ssl
creating build/lib.linux-x86_64-2.7/coro/emulation
copying coro/emulation/thread.py -> build/lib.linux-x86_64-2.7/coro/emulation
copying coro/emulation/threading.py -> build/lib.linux-x86_64-2.7/coro/emulation
copying coro/emulation/socket.py -> build/lib.linux-x86_64-2.7/coro/emulation
copying coro/emulation/init.py -> build/lib.linux-x86_64-2.7/coro/emulation
creating build/lib.linux-x86_64-2.7/coro/dns
copying coro/dns/reply.py -> build/lib.linux-x86_64-2.7/coro/dns
copying coro/dns/cache.py -> build/lib.linux-x86_64-2.7/coro/dns
copying coro/dns/stub_resolver.py -> build/lib.linux-x86_64-2.7/coro/dns
copying coro/dns/init.py -> build/lib.linux-x86_64-2.7/coro/dns
copying coro/dns/exceptions.py -> build/lib.linux-x86_64-2.7/coro/dns
creating build/lib.linux-x86_64-2.7/coro/clocks
copying coro/clocks/init.py -> build/lib.linux-x86_64-2.7/coro/clocks
creating build/lib.linux-x86_64-2.7/coro/asn1
copying coro/asn1/init.py -> build/lib.linux-x86_64-2.7/coro/asn1
creating build/lib.linux-x86_64-2.7/coro/ssh
copying coro/ssh/init.py -> build/lib.linux-x86_64-2.7/coro/ssh
creating build/lib.linux-x86_64-2.7/coro/db
copying coro/db/init.py -> build/lib.linux-x86_64-2.7/coro/db
creating build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/static_key_storage.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/openssh_authorized_keys.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/key_storage.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/dss.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/remote_host.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/public_private_key.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/rsa.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/openssh_key_storage.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/rebuild.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/openssh_key_formats.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/openssh_known_hosts.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
copying coro/ssh/keys/ber.py -> build/lib.linux-x86_64-2.7/coro/ssh/keys
creating build/lib.linux-x86_64-2.7/coro/ssh/transport
copying coro/ssh/transport/client.py -> build/lib.linux-x86_64-2.7/coro/ssh/transport
copying coro/ssh/transport/constants.py -> build/lib.linux-x86_64-2.7/coro/ssh/transport
copying coro/ssh/transport/transport.py -> build/lib.linux-x86_64-2.7/coro/ssh/transport
copying coro/ssh/transport/server.py -> build/lib.linux-x86_64-2.7/coro/ssh/transport
copying coro/ssh/transport/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/transport
creating build/lib.linux-x86_64-2.7/coro/ssh/l4_transport
copying coro/ssh/l4_transport/coro_socket_transport.py -> build/lib.linux-x86_64-2.7/coro/ssh/l4_transport
copying coro/ssh/l4_transport/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/l4_transport
creating build/lib.linux-x86_64-2.7/coro/ssh/scp
copying coro/ssh/scp/client.py -> build/lib.linux-x86_64-2.7/coro/ssh/scp
copying coro/ssh/scp/core.py -> build/lib.linux-x86_64-2.7/coro/ssh/scp
copying coro/ssh/scp/cli.py -> build/lib.linux-x86_64-2.7/coro/ssh/scp
copying coro/ssh/scp/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/scp
creating build/lib.linux-x86_64-2.7/coro/ssh/auth
copying coro/ssh/auth/userauth.py -> build/lib.linux-x86_64-2.7/coro/ssh/auth
copying coro/ssh/auth/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/auth
creating build/lib.linux-x86_64-2.7/coro/ssh/wrapper
copying coro/ssh/wrapper/coro_interactive_ssh_wrapper.py -> build/lib.linux-x86_64-2.7/coro/ssh/wrapper
copying coro/ssh/wrapper/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/wrapper
creating build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/interactive_session.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/constants.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/tty_modes.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/data_buffer.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/channel.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
copying coro/ssh/connection/connect.py -> build/lib.linux-x86_64-2.7/coro/ssh/connection
creating build/lib.linux-x86_64-2.7/coro/ssh/key_exchange
copying coro/ssh/key_exchange/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/key_exchange
copying coro/ssh/key_exchange/diffie_hellman.py -> build/lib.linux-x86_64-2.7/coro/ssh/key_exchange
creating build/lib.linux-x86_64-2.7/coro/ssh/compression
copying coro/ssh/compression/none.py -> build/lib.linux-x86_64-2.7/coro/ssh/compression
copying coro/ssh/compression/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/compression
creating build/lib.linux-x86_64-2.7/coro/ssh/util
copying coro/ssh/util/mpint.py -> build/lib.linux-x86_64-2.7/coro/ssh/util
copying coro/ssh/util/debug.py -> build/lib.linux-x86_64-2.7/coro/ssh/util
copying coro/ssh/util/password.py -> build/lib.linux-x86_64-2.7/coro/ssh/util
copying coro/ssh/util/packet.py -> build/lib.linux-x86_64-2.7/coro/ssh/util
copying coro/ssh/util/random.py -> build/lib.linux-x86_64-2.7/coro/ssh/util
copying coro/ssh/util/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/util
creating build/lib.linux-x86_64-2.7/coro/ssh/mac
copying coro/ssh/mac/hmac.py -> build/lib.linux-x86_64-2.7/coro/ssh/mac
copying coro/ssh/mac/hmac_sha1.py -> build/lib.linux-x86_64-2.7/coro/ssh/mac
copying coro/ssh/mac/hmac_md5.py -> build/lib.linux-x86_64-2.7/coro/ssh/mac
copying coro/ssh/mac/none.py -> build/lib.linux-x86_64-2.7/coro/ssh/mac
copying coro/ssh/mac/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/mac
creating build/lib.linux-x86_64-2.7/coro/ssh/cipher
copying coro/ssh/cipher/none.py -> build/lib.linux-x86_64-2.7/coro/ssh/cipher
copying coro/ssh/cipher/blowfish_cbc.py -> build/lib.linux-x86_64-2.7/coro/ssh/cipher
copying coro/ssh/cipher/init.py -> build/lib.linux-x86_64-2.7/coro/ssh/cipher
copying coro/ssh/cipher/des3_cbc.py -> build/lib.linux-x86_64-2.7/coro/ssh/cipher
creating build/lib.linux-x86_64-2.7/coro/db/postgres
copying coro/db/postgres/postgres.py -> build/lib.linux-x86_64-2.7/coro/db/postgres
copying coro/db/postgres/init.py -> build/lib.linux-x86_64-2.7/coro/db/postgres
file backdoor.py (for module backdoor) not found
file coro_process.py (for module coro_process) not found
file coro_unittest.py (for module coro_unittest) not found
running build_ext
cythoning coro/event_queue.pyx to coro/event_queue.cpp
building 'coro.event_queue' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/coro
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c coro/event_queue.cpp -o build/temp.linux-x86_64-2.7/coro/event_queue.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/coro/event_queue.o -o build/lib.linux-x86_64-2.7/coro/event_queue.so
cythoning coro/_coro.pyx to coro/_coro.c

Error compiling Cython file:

...
cdef _die (self)
cdef __interrupt (self, the_exception)
cdef int try_selfish (self)

choose a library for stack compression

IF COMPILE_LZ4:

^

coro/_coro.pxd:126:14: Compile-time name 'COMPILE_LZ4' not defined

Error compiling Cython file:

...
cdef int try_selfish (self)

choose a library for stack compression

IF COMPILE_LZ4:
include "zstack_lz4.pxd"
ELIF COMPILE_LZO:

^

coro/_coro.pxd:128:16: Compile-time name 'COMPILE_LZO' not defined

Error compiling Cython file:

...

def __dealloc__ (self):
    global live_coros
    live_coros = live_coros - 1
    if self.stack_copy != NULL:
        IF CORO_DEBUG:
                    ^

coro/_coro.pyx:241:25: Compile-time name 'CORO_DEBUG' not defined

Error compiling Cython file:

...
force was set to False).
:raises ScheduleError: Attempted to raise an exception on the currently
running coro.
:raises NotStartedError: The coroutine has not started, yet.
"""
IF CORO_DEBUG:

^

coro/_coro.pyx:557:21: Compile-time name 'CORO_DEBUG' not defined

Error compiling Cython file:

...

class WakeUp (Exception):
"""A convenience exception used to wake up a sleeping thread."""

choose a library for stack compression

IF COMPILE_LZ4:

^

coro/_coro.pyx:865:14: Compile-time name 'COMPILE_LZ4' not defined

Error compiling Cython file:

...
"""A convenience exception used to wake up a sleeping thread."""

choose a library for stack compression

IF COMPILE_LZ4:
include "zstack_lz4.pyx"
ELIF COMPILE_LZO:

^

coro/_coro.pyx:867:16: Compile-time name 'COMPILE_LZO' not defined

Error compiling Cython file:

...
self._last.compressed = False
#2) get some storage
if self._last.stack_size != size:
# XXX: more heuristics to avoid malloc
if self._last.stack_copy:
IF CORO_DEBUG:

^

coro/_coro.pyx:939:33: Compile-time name 'CORO_DEBUG' not defined

Error compiling Cython file:

...
:raises TimeoutError: The function did not return within the specified
timeout.
"""
cdef timebomb tb
cdef event e
IF CORO_DEBUG:

^

coro/_coro.pyx:1081:21: Compile-time name 'CORO_DEBUG' not defined

Error compiling Cython file:

...
"""Sleep until a specific point in time.

    :param when: The TSC value when you want the coroutine to wake up.
    """
    cdef event e
    IF CORO_DEBUG:
                ^

coro/_coro.pyx:1113:21: Compile-time name 'CORO_DEBUG' not defined

Error compiling Cython file:

...
return self._waiting.size

cdef _wait (self):
    cdef coro me
    me = the_scheduler._current
    IF CORO_DEBUG:
                ^

coro/sync.pyx:609:21: Compile-time name 'CORO_DEBUG' not defined

Error compiling Cython file:

...

extras used by ironport only

include "ironport.pyx"
include "profile.pyx"
IF UNAME_SYSNAME == "FreeBSD":
include "aio.pyx"
IF COMPILE_LIO:

^

coro/_coro.pyx:1302:14: Compile-time name 'COMPILE_LIO' not defined

Error compiling Cython file:

...
]

IF UNAME_SYSNAME == "FreeBSD":
versions.append (aio_version)

IF COMPILE_LIO:

^

coro/_coro.pyx:1320:14: Compile-time name 'COMPILE_LIO' not defined

building 'coro._coro' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/home/syadasti/shrapnel/. -I/home/syadasti/shrapnel/include -I/usr/include/python2.7 -c coro/_coro.c -o build/temp.linux-x86_64-2.7/coro/_coro.o -Wno-unused-function
coro/_coro.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
error: command 'gcc' failed with exit status 1
syadasti@charlie:~/shrapnel$

The decoder fun for this is set to 'int', it needs to be float.

Repro:

create table thing (mynum numeric (3,2));
insert into thing (mynum) values (1.0);
select * from thing;

I have tried to execute that example: https://github.com/ironport/shrapnel/blob/master/coro/http/demo/jsonrpc_server.py and got error:

$ python env/http_server.py
FIX ber.py to use coro.asn1.ber
Segmentation fault

Then I tried run in gdb:

(gdb) run
Starting program: /str/development/projects/open-source/shrapnel/env/bin/python env/http_server.py
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
FIX ber.py to use coro.asn1.ber

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff675e875 in _wrap1 () from /str/development/projects/open-source/shrapnel/env/lib/python2.7/site-packages/coro-1.0.3_000-py2.7-linux-x86_64.egg/coro/_coro.so
(gdb) c
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)

As written, the bind address is passed in to the start() method rather than the constructor. This makes it very difficult to cleanly start up multiple servers on multiple addresses. I'm just announcing my intention to move the addr param to the constructor soon. [I've already done it on a personal branch].

A library from google that sounds competitive with lzo: http://code.google.com/p/snappy/

When we send kexinit to the server, we give it a list of supported host key types. If we have e.g. an RSA key in our known_hosts file, but do not place RSA at the front of the list, then we'll get an Invalid_Server_Public_host_Key because the server always offers DSS first. (apparently there's no retry?)

log:

root@charlie:# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.3 LTS"
root@charlie:# uname -a
Linux charlie 3.5.0-40-generic #62precise1-Ubuntu SMP Fri Aug 23 17:38:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
root@charlie:# python --version
Python 2.7.3
root@charlie:# pip --version
pip 1.0 from /usr/lib/python2.7/dist-packages (python 2.7)
root@charlie:# pip install coro
Downloading/unpacking coro
Downloading master (unknown size): 479Kb downloaded
Cannot determine compression type for file /tmp/pip-aM4spZ-unpack/master
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main
self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 223, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 955, in prepare_files
self.unpack_url(url, location, self.is_download)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1072, in unpack_url
return unpack_http_url(link, location, self.download_cache, only_download)
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 447, in unpack_http_url
unpack_file(temp_location, location, content_type, link)
File "/usr/lib/python2.7/dist-packages/pip/util.py", line 468, in unpack_file
untar_file(filename, location)
File "/usr/lib/python2.7/dist-packages/pip/util.py", line 435, in untar_file
shutil.copyfileobj(fp, destfp)
File "/usr/lib/python2.7/shutil.py", line 49, in copyfileobj
buf = fsrc.read(length)
AttributeError: 'NoneType' object has no attribute 'read'

Storing complete log in /root/.pip/pip.log

root@charlie:~# cat .pip/pip.log

/usr/bin/pip run on Tue Sep 10 20:23:41 2013
Downloading/unpacking coro
Getting page http://pypi.python.org/simple/coro
URLs to search for versions for coro:

• http://pypi.python.org/simple/coro/
  Getting page http://pypi.python.org/simple/coro/
  Analyzing links from page https://pypi.python.org/simple/coro/
  Skipping link http://github.com/ironport/shrapnel (from https://pypi.python.org/simple/coro/); not a file
  Found link http://github.com/ironport/shrapnel/tarball/master#egg=coro-1.0.2 (from https://pypi.python.org/simple/coro/), version: 1.0.2
  Found link http://github.com/samrushing/shrapnel/tarball/master#egg=coro-1.0.1 (from https://pypi.python.org/simple/coro/), version: 1.0.1
  Found link http://github.com/ironport/shrapnel/tarball/master#egg=coro-1.0.1 (from https://pypi.python.org/simple/coro/), version: 1.0.1
  Using version 1.0.2 (newest of versions: 1.0.2, 1.0.1, 1.0.1)
  Downloading from URL http://github.com/ironport/shrapnel/tarball/master#egg=coro-1.0.2 (from https://pypi.python.org/simple/coro/)
  Cannot determine compression type for file /tmp/pip-aM4spZ-unpack/master
  Exception:
  Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main
  self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 223, in run
  requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 955, in prepare_files
  self.unpack_url(url, location, self.is_download)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1072, in unpack_url
  return unpack_http_url(link, location, self.download_cache, only_download)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 447, in unpack_http_url
  unpack_file(temp_location, location, content_type, link)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 468, in unpack_file
  untar_file(filename, location)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 435, in untar_file
  shutil.copyfileobj(fp, destfp)
  File "/usr/lib/python2.7/shutil.py", line 49, in copyfileobj
  buf = fsrc.read(length)
  AttributeError: 'NoneType' object has no attribute 'read'
  root@charlie:~#

Python 2.7
Cython 0.20.1

$ python setup.py build

Error compiling Cython file:
...
cdef int scope_id
cdef addrinfo * ai
if len(address) == 4:
# as per python return value from getaddrinfo() and arg to connect()
ip, port, flowinfo, scope_id = address
sin6.sin6_flowinfo = htonl (flowinfo)
^
coro/socket.pyx:830:16: Object of type 'sockaddr_in6' has no attribute 'sin6_flowinfo'

and similar errors for 'sin6_scope_id' eventually terminating in:

building 'coro._coro' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/coro
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/home/venzen/sourcecode/shrapnel/. -I/home/venzen/sourcecode/shrapnel/include -I/usr/include/python2.7 -c coro/_coro.c -o build/temp.linux-x86_64-2.7/coro/_coro.o -Wno-unused-function
coro/_coro.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
#error Do not use this file, it is the result of a failed Cython compilation.
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

_Using Python 3.4 results in errors with building package distribute egg

Notice that there's no way of running the 'loop' more than once. Can anyone explain this?
I think the loop should just be removed, unlike read/write, you can't call connect() more than once if it blocks.

The default of max=0 works, but specifying any other value for max results in no accepted sockets being returned in the list. - no return at all.

I hacked around the issue for my purposes by editing the socket.pyx in the coro subdir, and in the accept_many cpdef,

(commenting out:)

result = PyList_New(max)

change:
if max == 0:
result.append (element)
else:
#result[count] = element
result.append (element)
count = count + 1

while this works for my purposes, I'm sure its not what was intended and someone more knowledgeable could provide a proper fix.

The (new) HTTP server needs some improvements to how it logs. It needs to concentrate all logging into the request.done() method, and ensure that it is always called. Currently, there are sprinklings of log calls in several places, probably because of the error exit path. [for example, a connection reset will cause an exception in request.error() that keeps done() from ever being called]. This also means the format of the log line is unreliable and no information about bytes transferred is available on connections that have errors.