loolmeh / queues Goto Github PK

Attached is a tested and ready Beanstalkd backend. It requires beanstalkc.

I'd like to add a socket timeout to the Redis queue.  Here's a diff:

https://gist.github.com/9de7323fae027fde59a1

I imagine using the name redis would be more user friendly than using redisd 
for the redis backend name.

I propose renaming it or aliasing it to redis for new users.

I can send a patch if this is something you think is reasonable.

From IRC:

conley: matt_c: Also, you may be aware, but in memcache.py if you hit errors, 
they are not caught 
correctly because you have not imported memcache to be able to except 
memcache.MemcachedKeyError

The API itself as implemented by several lightweight queues is supported but 
you can't use 
memcached directly.  Update docs to reflect.

What steps will reproduce the problem?
1. Install current beanstalkd 1.4.6
2. Install queues 0.6.1
3. Attempt to write and read to the queue

I believe the backend needs to be updated to use the .watch(NAME) method on the 
beanstalk connection, and I think the reads also need to use the timeout to 
beanstalk's reserve() method, so that the backend can return None if there is 
nothing in the queue rather than waiting for something to arrive.

A blocking read is often useful when dealing with a queue. The consumer most 
times should sleep until a message becomes available.

This might be possible in other backends as well, I know Redis has a blpop 
method for just this purpose. The Queue.read() method should either take an 
optional blocking param (ignored in backends that don't support it) or a 
blocking_read() method should be added.

What steps will reproduce the problem?
>>> import os
>>> os.environ['QUEUE_BACKEND'] = 'redis'
>>> from queues import queues
>>> q = queues.Queue('foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'Queue'

What is the expected output? What do you see instead?

I would expect an error when importing queues.queues. If it can't find that 
backend, I should know about it explicitly.

Something like what happens if you don't specify QUEUE_REDIS_CONNECTION.

What version of the product are you using? On what operating system?

trunk, Linux.

Please provide any additional information below.

I will send a patch if you agree that this should be changed.

What steps will reproduce the problem?
1. Attempt to put a unicode message with beanstalkc backend

What version of the product are you using? On what operating system?
Latest versions of each, as at September 2011.

Please provide any additional information below.

See https://github.com/earl/beanstalkc/blob/master/beanstalkc.py#L124 - I think 
queues needs to convert messages to str before put. I have changed line 46 of 
queues/backends/beanstalkd.py from

                return self._connection.put(message)

to

                return self._connection.put(str(message))

http://github.com/disturbyte/zenqueue is a rather nice ultra-light-weight 
Python queue server 
that speaks JSON.

First of all, thanks for this software.

I have attached a patch which adds support for using a UNIX socket in the redis 
backend. This works with the redis-py library.

BTW I'm maintaining a friendly fork that contains this change if anyone is 
interested:
https://bitbucket.org/bgneal/queues

In the redis backend, in Queue.__init__ kwargs are prepared, but never passed 
to _get_connection(). The kwargs are again prepared within _get_connection(). 
The duplication should be removed, it was confusing to me at first.

http://code.google.com/p/queues/source/browse/trunk/queues/backends/redisd.py#40
http://code.google.com/p/queues/source/browse/trunk/queues/backends/redisd.py#50

I would suggest removing one of these two duplicated sets of code.

I can send a patch if you think this is reasonable.

The dummy backend still has a reference to a variable that went away in the 
last commit. Patch 
attached.

For testing purposes while using queues, this might help.

patch to implement a queue for each name for the dummy backend