It seems that WebSocketHandler.connectionLost is called only when client side call websocket.close().

So, If client side close browser window without calling websocket.close(), WebSocketHandler.connectionLost never be called.

At the moment HTTP PUT/POST body is kept and handled in RAM.

this represent a big limit:

• as uploaded data is limited to available RAM memory.
• RAM memory is overused in case of multiple uploads.

A possible solution can be to modify cyclone.httpserver.HTTPConnection._on_header to use TemporaryFile()
151 content_length = int(headers.get("Content-Length", 0))
152 if content_length:
153 if headers.get("Expect") == "100-continue":
154 self.transport.write("HTTP/1.1 100 (Continue)\r\n\r\n")
155 if content_length < 100000:
156 self._contentbuffer = StringIO()
157 else:
158 self._contentbuffer = tempfile.TemporaryFile()
159
160 self.content_length = content_length
161 self.setRawMode()
162 return

By doing (as done in Twisted) uploaded data is kept on filesystem and can be loaded in RAM only when needed using self._contentbuffer.read()

Issue found while developing Globaleaks and trying to upload file in the order of GB.

Hey again guys :)

I've been mucking around with moving our app over to Cyclone. I'm loving being back on twisted :)

One thing that is a problem for me... the httpclient.fetch returns a very sparse subset of the data available on the HTTPClientFactory... which makes it very hard to check errors properly, especially when dealing with the twitter api .. it's horrible :)

I don't want to use the curl based client from tornado though.

So.. I could write something to provide a similar or same level of information... I'm just wondering if you've considered using the Agent? http://twistedmatrix.com/documents/10.1.0/web/howto/client.html

The RequestHandler.write() method sets the Content-Type header to "text/javascript; charset=UTF-8" when JSONifying a dictionary. This should be "application/json" per RFC 4627.

hi,
There is a small problem with the new websocket support.
When I send Chinese characters through websocket, it will raise an exception:

Traceback (most recent call last):
   ..............(useless logs).........
    handler._execute(transforms, *args, **kwargs)
  File "/Users/daniel/Sites/nli_env/lib/python2.6/site-packages/cyclone-0.7-py2.6.egg/cyclone/web.py", line 1120, in _execute
    self.connectionMade(*args, **kwargs)
  File "/Users/daniel/Downloads/Dev/breeze/breeze/views/order.py", line 24, in connectionMade
    self.sendMessage("测试")
  File "/Users/daniel/Sites/nli_env/lib/python2.6/site-packages/cyclone-0.7-py2.6.egg/cyclone/web.py", line 949, in sendMessage
    newFrame += message.encode()
exceptions.UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

(FYI) It works fine when I change the file web.py, line 949 to:
newFrame += message.encode('utf-8')

please update the driver with the most recent txredisapi.

Hey guys, I can't fork because I already have a tornado fork!

Anyway, I'm mucking around with moving our app to Cyclone.. I've got one small fix.. I'll report any others I find.

line 247 in cyclone/auth.py needs a "d =" to resolve the variable issue on the next line :)

I hope that Cyclone Httpclient could support proxy like Tornado httpclient

• proxy_host
• proxy_port
• proxy_username
• proxy_password

I want pip install!

Currently points to 0.4

Cyclone can generate expires date for cookie https://github.com/fiorix/cyclone/blob/master/cyclone/web.py#L412

Example of generated date is: Thu, 31 Oct 2013 15:36:21 GMT

By rfc 2109 http://www.ietf.org/rfc/rfc2109.txt (10.1.2 Expires and Max-Age
) date should be Wdy, DD-Mon-YY HH:MM:SS GMT (%a, %d-%b-%Y %X GMT in python).

So, right expires should be: Thu, 31-Oct-2013 15:36:21 GMT

I want to set up a tiny fire and forget HTTP server that requires Digest Auth. The examples, however, only have a Basic Auth. So it'd be cool if there was a self contained example of how to run a server that does Digest Auth.

I have installed cyclone on ubuntu with Python 2.7. All seems fine from the command line as I can import cyclone packages, but when I try to use the scaffolding and run the created file 'start.sh' I get "unknown command: cyclone". I've messed around with PYTHONPATH a bit but still have this issue. Anyone else have a similar problem that they've worked through?

Hi! Is there any plans to support twisted.cred in Cyclone? Thanks!

When updating cyclone to 1.0 from any previous release a warming "Failed to update Twisted plugin cache." appears and, accordingly, it is impossible to use the cyclone plugin, because the plugin is not available.
Tested on centos 5, python 2.6.8, twisted 12.3.0, pyopenssl 0.12 installed.

Updated today to 1.0-rc10 from rc8.

Redis issues appeared. I am pretty much using RedisMixin from the demo.
With rc10 I am getting bizarre messages e.g.:
*3
$7
message
$54
notification.peer.0847b249-6af1-42f2-ae41-d2bc972e6576
$1

cyclone.jsonrpc.JsonrpcRequestHandler forces request id to be int:

jsonid = req["id"]
assert isinstance(jsonid, types.IntType), \
                           "Invalid id type: %s" % type(jsonid)

But http://json-rpc.org/wiki/specification says:
id - The request id. This can be of any type. It is used to match the response with the request that it is replying to.

So I suppose there is no need in the assert above.

Code from http://cyclone.io/ Code samples in Twisted Application Style simply does not work.

There's an error in last line, instead of internet.TCPServer(8888, foobar(), interface="127.0.0.1").setServiceParent(application) it should be internet.TCPServer(8888, foobar, interface="127.0.0.1").setServiceParent(application)

The httpserver.py file in cyclone imports fcntl , and this is not used anywhere in the whole code. this prevents it from working on windows. When i uncomment it, all works good. can you please check?

The RequestHandler._handle_request_exception misses cyclone.web.HTTPError's emebedded in twisted.internet.defer.FirstError's, which can result from using twisted.internet.defer.DeferredList's created with "fireOnOneErrback=True".

Here's a simple example of how this might happen::

import cyclone.web
from twisted.internet.defer import inlineCallbacks, gatherResults
...
class MyHandler(cyclone.web.RequestHandler):
def get(self):
return self.some_method(...)
@inlineCallbacks
def some_method(self, _args, *_kw):
d1 = async_call_1(...)
d2 = async_call_2(...)
r1, r2 = yield gatherResults([d1, d2])
...
If the deferred "d1" above, for instance, throws a cyclone.web.HTTPError, it will be wrapped in a twisted.python.failure.Failure instance which will be wrapped in a twisted.internet.defer.FirstError exception.

The following 5 lines of code inserted at the beginning of the RequestHandler._handle_request_exception method worked for me:

def _handle_request_exception(self, e):
try:
if isinstance(e.value, defer.FirstError):
e = e.value.subFailure
except:
pass
...

Thanks for all of your work on Cyclone. I work for Lucasfilm (we're heavy Twisted users) and we're starting to use cyclone and really liking it.

Cheers!
Ryan

If an attacker is able to successfully land an XSS payload, he may be able to construct POST requests to arbitrary URIs in your web application. However, without access to the XSRF token to include in the form submission, these POST requests will be rejected. Putting the XSRF cookie out of reach of the JavaScript payload via the HttpOnly attribute hampers these types of attacks.

See https://github.com/fiorix/cyclone/blob/master/cyclone/httpserver.py#L260.

Is there any particular reason this isn't used, and 'http' is used unless its behind a proxy and using xheaders?

The Constructor of cyclone.sse.SSEHandler does not accept **kwargs (to be passed to initialize method):

class SSEHandler(RequestHandler):
    def __init__(self, application, request):
        RequestHandler.__init__(self, application, request)
        ...

Should IMHO look like this:

class SSEHandler(RequestHandler):
    def __init__(self, application, request, **kwargs):
        RequestHandler.__init__(self, application, request, **kwargs)
        ...

What the purpose of "secure cookies" setting https://github.com/fiorix/cyclone/blob/master/cyclone/web.py#L458? As I understand it's only purpose is to allow or not secure cookies. But I don't see any usage of this setting in code.

Are you considering supporting SocketIO for cross browser server to client notifications?

twisted is a large project with many dependencies and not all of it's services are consumed by cyclone, however, how much of it is actually used?

env: lastest build of cyclone.
chrome 17.0.932.0 dev

run websocket demo.
visit http://localhost:8888, it raises errors like "Unexpected response code: 403"

nothing happened on the console. no log at all.

while it DOES work with Safari Version 5.1.1 (6534.51.22).

does it have anything to do with the Chrome local security issue?

exceptions.TypeError: Header entry 'Content-Type' should be list but found instance of <type 'str'> instead

That's a little confusing, since it's not my 'Content-Type' str that should be a list. It's the dict value that goes with that key. And the dict key isn't a header entry itself, it's the header name to be used on potentially multiple header entries. Here's my attempt at a clearer message:

'Content-Type' key should have a value that is a list of header values, not a <type 'str'>

I wanted to add some custom SMTP headers for the mails I generate but there seems to be no option to do that. So I have just added a function to add headers to the mail.
Hope it will be useful for all.

In cyclone.web, line 1113 func _load_ui_modules there's a typo:

for m in list: self._load_ui_modules(m)

should be

for m in modules: self._load_ui_modules(m)

list is a type :)

fixing that makes it load correctly a list of uimodules.
Otherwise a TypeError is raised.

regards,
mat

If I pass URL including non-ascii character, such as http://テスト/, escape.linkify crash.

>>> import cyclone.escape
>>> cyclone.escape.linkify(u'http://テスト/')
Traceback (most recent call last):
  File "", line 1, in 
  File "cyclone/escape.py", line 302, in linkify
    return _URL_RE.sub(make_link, text)
  File "cyclone/escape.py", line 296, in make_link
    (href, params, url)).decode("unicode_escape")
UnicodeEncodeError: 'ascii' codec can't encode characters in position 16-18: ordinal not in range(128)

Do you guys think it's worth keeping cyclone as a fork of tornado? I'd love to maintain forks of both... I definitely can't drop my fork of tornado, but github won't allow me to also fork cyclone because it simply sees it as a fork of tornado!

I guess this is more of a github issue, but wanted to float it with you first :)

Hello guys,

I'm having problems rewriting the JsonrpcRequestHandler to my needs. What I want to do is define the jsonrpc methods in an different module. Therefore modified my JsonrpcRequestHandler like this (simplified version):

class InternalApiHandler(cyclone.jsonrpc.JsonrpcRequestHandler):
    def initialize(self, mongodb):
        self.mongodb = mongodb

    def auth_and_process_internal_call(fn):
        """ wrapper method for jsonrpc methods """
        @defer.inlineCallbacks
        @functools.wraps(fn)
        def wrapper(self, *args, **kwargs):
            response = yield defer.inlineCallbacks(fn)(self, *args, **kwargs)
            defer.returnValue(response)
        return wrapper

    @auth_and_process_internal_call
    def call_internal(self, method_name, *args, **kwargs):
        logging.info("INTERNAL CALL: '%s'", method_name)
        response = yield getattr(self.mongodb, method_name)(*args, **kwargs)
        defer.returnValue(response)

    def __getattr__(self, name):
        if name.startswith("jsonrpc_"):
            method = getattr(self.mongodb, name[len("jsonrpc_"):])
            if callable(method):
                return functools.partial(self.call_internal, name[len("jsonrpc_"):])
        else:
            return getattr(self, name)

The mongodb class looks like this:

class DB(object):
    @defer.inlineCallbacks
    def list_users(self):
        yield
        defer.returnValue("test")

Running this works for me and I get "test" as response. However I'm also getting this exception on the cyclone side:

Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.AttributeError'> ignored

I'm not really sure what causes this. Help appreciated.

cheers
Hannes

Any thoughts on uploading to pypi, or simply rolling a release in the downloads section of github?

Is there a way to retain the original stack trace in the error logs? It's very difficult to debug controllers.

I'm seeing very strange behavior on empty POST requests. From the client side, it looks like the server simply never responds. But in the server output log, it looks like it is transformed into a GET. The incoming pull request fixes this.

It looks like autoreload was commented from the original tornado code. What would it take to get it going again in twisted?

Could we add a route or request method to request handler that accepts all requests for all HTTP methods? Or is this already there?

class Handler(web.RequestHandler):
    def route(self):
        # Do some proxy work, etc.
        pass

For bottle-style routing... would anyone be opposed to allowing ANY to be specified as the method?

from cylcone.bottle import route

@route(r'^/path/goes/here', method='ANY')
def request(web):
    # Do some proxy work, etc.
    pass

This does exist in bottle; just not documented (as far as I can tell). See: https://github.com/bottlepy/bottle/blob/master/bottle.py#L405.

I tried inlineCallback/returnValue or deferredGenerator/waitForDeferred, didn't work.
Any idea?

decorator @removeslash in cyclone.web may remove final '/' if not expected. would be nice use it, but the Cyclone code check only HEAD and GET while we need in http://github.com/globaleaks/GLBackend checks in a complete CURD .

Would it possible to extend https://github.com/fiorix/cyclone/blob/master/cyclone/web.py of line 1090 also to POST and DELETE ?

doing a:

"pip install -r requirements.txt"

and having:

cyclone==1.1
Twisted==13.1.0

causes:

Downloading/unpacking Twisted==13.1.0 (from -r requirements.txt (line 1))
  Running setup.py egg_info for package Twisted

Downloading/unpacking cyclone==1.1 (from -r requirements.txt (line 2))
  Running setup.py egg_info for package cyclone

    package init file 'twisted/plugins/__init__.py' not found (or not a regular file)
    *** Failed to update Twisted plugin cache. ***
    No module named twisted.plugin

I'm about to upgrade the redis client shipped with cyclone.
Connection method names have changed, and it's now a single file - as opposed to a directory with many files in it.

https://github.com/fiorix/txredisapi

To avoid breaking whatever you have deployed out there, I'll include the old connection methods and leave it there for a while.
But, please consider updating to the new names. (i.e.: from cyclone.redis.lazyRedisConnectionPool to cyclone.redis.lazyConnectionPool).

The new version of txredisapi is the real deal, with support for redis transactions.

Greetings, Alexandre!

Thank you very much for cyclone! It is really cool!

But i have a trouble and a note:)

[Trouble] Making cyclone-sse project (https://github.com/FZambia/cyclone-sse) i came across a problem - can not install cyclone on CentOS 5. This is because their is no OpenSSL 0.9.8f distribution. Here detail description http://stackoverflow.com/questions/7340784/easy-install-pyopenssl-error
In my project i avoided this problem by adding OS version check in setup.py and using pyopenssl==0.12 in case of Centos 5.

requires = ["twisted", "cyclone", "txAMQP"]

try:
    # to avoid installation problems on centos 5
    # which has no distributions for OpenSSL 0.9.8f
    from distutils.version import StrictVersion
    import platform
    os, version, _type = platform.linux_distribution()
    if os == "CentOS" and StrictVersion(version) < StrictVersion('6.0'):
        requires.insert(0, "pyopenssl==0.12")
except:
    pass

But standalone cyclone still can not be installed

[Note] While installing i got following warning:

package init file 'twisted/plugins/__init__.py' not found (or not a regular file)
*** Failed to update Twisted plugin cache. ***

Thanks for your time!

When setting the static_path setting in windows, the result is always HTTP code 403.
Here is the patch that fixes it:

1341c1341
<         self.root = os.path.abspath(path + "/") 

---
>        self.root = os.path.abspath(path) + "/"

Seems like it is trying to open a zipfile in a non-binary mode in app.py.
This happens when trying to create a new project like this:

python -m cyclone.app -p newProject

Just including "rb" as the mode seems to fix this.

Using:
Windows 7 64-bit
Python 2.7.2 and 2.7.3

I have a dependency on Cyclone in my setup.py file. When I run python setup.py develop (in a virtualenv) I get the following error: Unexpected error while writing cache file

Full output below.

Searching for cyclone==1.0-rc13
Reading http://pypi.python.org/simple/cyclone/
Reading http://cyclone.io/
Reading http://github.com/fiorix/cyclone/
Reading http://github.com/fiorix/cyclone
Best match: cyclone 1.0-rc13
Downloading http://pypi.python.org/packages/source/c/cyclone/cyclone-1.0-rc13.tar.gz#md5=6b7670a9912865d4eb0601eac5ca0363
Processing cyclone-1.0-rc13.tar.gz
Running cyclone-1.0-rc13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-HJ6ITW/cyclone-1.0-rc13/egg-dist-tmp-wmfvYj
package init file 'twisted/plugins/__init__.py' not found (or not a regular file)
zip_safe flag not set; analyzing archive contents...
cyclone.app: module references __file__
Unexpected error while writing cache file
Traceback (most recent call last):
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/sandbox.py", line 72, in run
    return func()
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/sandbox.py", line 32, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 70, in <module>
    list(getPlugins(IPlugin))
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/plugin.py", line 209, in getPlugins
    allDropins = getCache(package)
--- <exception caught here> ---
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/plugin.py", line 181, in getCache
    dropinPath.setContent(pickle.dumps(dropinDotCache))
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/filepath.py", line 1247, in setContent
    f = sib.open('w')
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/filepath.py", line 775, in open
    return self.create()
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/python/filepath.py", line 1282, in create
    fdint = os.open(self.path, _CREATE_FLAGS)
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/sandbox.py", line 231, in open
    self._violation("os.open", file, flags, mode)
  File "/home/minion/minion/env/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/sandbox.py", line 185, in _violation
    raise SandboxViolation(operation, args, kw)
setuptools.sandbox.SandboxViolation: SandboxViolation: os.open('/home/minion/minion/env/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/plugins/BRzD3HAWjLoz_1cBdropin.cache.new', 194, 511) {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.
Adding cyclone 1.0-rc13 to easy-install.pth file

Installed /home/minion/minion/env/lib/python2.7/site-packages/cyclone-1.0_rc13-py2.7.egg

def get(self):
        self.set_header("content-type", "application/xhtml+xml") # quietly ignored; output header is text/html

def get(self):
        self.set_header("Content-Type", "application/xhtml+xml") # succeeds

I think the first one should work just like the second one, since http headers are case insensitive.

There appears to be a leak of sorts with Keep-Alive requests. It appears this can be recreated simply by using the "Hello, World" example Cyclone app and running ab. Without keep alive using this command:

ab -c 5 -n 10000 http://192.168.2.1:8888/

The results are as expected. First entry is before running Apachebench, the second is after:

bshelton 31058 0.0 0.2 113828 14528 pts/23 S+ 10:51 0:00 /usr/bin/python2.6 hello-world.py
bshelton 31058 4.6 0.2 113984 14688 pts/23 S+ 10:51 0:05 /usr/bin/python2.6 hello-world.py

Enabling Keep-Alive in ab using this command:

ab -k -c 5 -n 10000 http://192.168.2.1:8888/

Yields far different results:

bshelton 31158 10.0 0.2 113828 14528 pts/23 S+ 10:54 0:00 /usr/bin/python2.6 hello-world.py
bshelton 31158 24.3 1.2 173984 74672 pts/23 S+ 10:54 0:04 /usr/bin/python2.6 hello-world.py

This will continue, unbounded, until no more requests are received. Peculiarly, the garbage collector will appear to free a small chunk of this after an indeterminate amount of time but it will never free all of it.

I poked around internally using Twisted's manhole feature and some permutations of objgraph, pympler, and heapy, and have come to the conclusion that it appears Keep-Alive requests fail to fire code necessary to fully release all references to cyclone.httpserver.HTTPRequest objects created per request. Consequently, the requests pile up and are never freed. (Actually, this isn't true. Near as I can tell, it does appear that the gc eventually frees them, but the str, dict, and list objects referencing them/referenced by them never appear to clear up. I don't know enough about Python internals to understand what is happening in this particular case since the HTTPRequest objects do seem to disappear if and only if no requests are received by cyclone after some length of time.)

For what it's worth, this was tested on Gentoo Linux using Python 2.6.2 and 2.6.5 running on x86 and amd64, respectively. Both test instances were using Twisted 10.1 and with the Cyclone 0.4 package and, later, directly from the github repo trunk.

The problem:

When running in windows the following error is received:

Traceback (most recent call last): File "webmain.tac", line 4, in import cyclone.web File "C:\Python27\lib\site-packages\cyclone\web.py", line 21, in from cyclone import __version__, escape, template, httpserver File "C:\Python27\lib\site-packages\cyclone\httpserver.py", line 24, in import fcntl ImportError: No module named fcntl

Proposed solution

it seems that fcntl is not required in httpserver since there are no refernces to it.
removing it solved the problem

redis module is outdated.
Maybe redis module should be linked to txredisapi.

Hi I noticed there is a problem in the last commit c4ae24d

•    d = fetch(
  

  •        self.__rpcUrl,
    

  •        method="POST",
    

  •        postdata=q,
    

  •        headers={
    

  •            "Content-Type": "application/json-rpc"
    

  •        }
    

  •    )
    

"Content-Type": "application/json-rpc": ought to be

"Content-Type": ["application/json-rpc"]

otherwise , I got exception.

Thanks.