seatgeek / amqp-dispatcher Goto Github PK
View Code? Open in Web Editor NEWA daemon to run AMQP consumers
amqp-dispatcher's People
Contributors
Stargazers
Watchers
Forkers
zackkitzmiller cce rickhanlonii shensiduanxing jxav soul0517 vaibhavkaul ekelleyv b-r-oleary gajohnson samuel-b-lee rafalstapinskiamqp-dispatcher's Issues
Use as a library
Make it easy to include the dispatcher with specific queues via a library call so that it can be included in scripts that shouldn't run forever.
Support calling function to create consumer
This would allow constructor functions using dependency injection to simplify testing.
Might not reconnect after RabbitMQ node shutdown
Just throws an error without a trace
local variable 'connection' referenced before assignment
Prefetch count should be separate from worker count
And worker counts of 0 should not consume from a queue. At the moment settings a prefetch of 0 in order to get 0 workers will cause all messages to be delivered (since that what prefetch 0 does)
Allow a pool to consumer mapping
In order to limit the use of constrained resources it would be nice to say a single consumer is responsible for multiple queues or allow a single pool between multiple consumers (which may or may not be the same resource).
Adding any kind of pool configuration above number of workers may be a bit annoying. Just allowing multiple consume commands called for a single consumer seems like a safe bet and much easier.
- consumer: example.consumer:Consumer
queues:
- queue: example_queue_1
prefetch_count: 2
- queue: example_queue_2
prefetch_count: 4
Exits 0 with stack trace
[2013-03-01 14:07:33] malone - INFO - Response: 200: u'{}'
* using prod settings
Traceback (most recent call last):
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
result = self._run(*self.args, **self.kwargs)
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/amqpdispatcher/dispatcher.py", line 195, in message_pump_greenthread
connection.read_frames()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 354, in read_frames
self._transport.process_channels( p_channels )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/transports/transport.py", line 38, in process_channels
channel.process_frames()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/channel.py", line 206, in process_frames
self.dispatch( frame )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 435, in dispatch
cb( frame )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 471, in _recv_start
self._send_start_ok()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 480, in _send_start_ok
self.send_frame( MethodFrame(self.channel_id, 10, 11, args) )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/channel.py", line 256, in send_frame
self._connection.send_frame(frame)
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 397, in send_frame
self._transport.write( buf )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/transports/gevent_transport.py", line 99, in write
return super(GeventTransport,self).write(data)
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/transports/socket_transport.py", line 121, in write
self.connection.transport_closed( msg='error writing to %s'%(self._host) )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 244, in transport_closed
self._callback_close()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 319, in _callback_close
self._close_cb()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/amqpdispatcher/dispatcher.py", line 33, in connection_closed_cb
connection.close_info,)
UnboundLocalError: local variable 'connection' referenced before assignment
<Greenlet at 0x9f8925c: message_pump_greenthread(<haigha.connections.rabbit_connection.RabbitConnec)> failed with UnboundLocalError
Exits 0 with stack trace
Traceback (most recent call last):
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
result = self._run(*self.args, **self.kwargs)
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/amqpdispatcher/dispatcher.py", line 195, in message_pump_greenthread
connection.read_frames()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 354, in read_frames
self._transport.process_channels( p_channels )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/transports/transport.py", line 38, in process_channels
channel.process_frames()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/channel.py", line 206, in process_frames
self.dispatch( frame )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 435, in dispatch
cb( frame )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 471, in _recv_start
self._send_start_ok()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 480, in _send_start_ok
self.send_frame( MethodFrame(self.channel_id, 10, 11, args) )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/channel.py", line 256, in send_frame
self._connection.send_frame(frame)
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 397, in send_frame
self._transport.write( buf )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/transports/gevent_transport.py", line 99, in write
return super(GeventTransport,self).write(data)
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/transports/socket_transport.py", line 121, in write
self.connection.transport_closed( msg='error writing to %s'%(self._host) )
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 244, in transport_closed
self._callback_close()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/haigha/connection.py", line 319, in _callback_close
self._close_cb()
File "/apps/production/seatgeek/virtualenvs/malone/local/lib/python2.7/site-packages/amqpdispatcher/dispatcher.py", line 33, in connection_closed_cb
connection.close_info,)
UnboundLocalError: local variable 'connection' referenced before assignment
<Greenlet at 0x8e8925c: message_pump_greenthread(<haigha.connections.rabbit_connection.RabbitConnec)> failed with UnboundLocalError
Gracefully fail on import error
If you add a consumer with an invalid module name, the dispatcher should only fail that consumer.
Connect to multiple RabbitMQ servers
Allow connections to multiple rabbitmq servers. This would allow consuming from multiple clusters to drain an old node before terminating it.
Set the hostname on the consumer tag
Not currently set properly on consumers, instead has some sort of hash.
Test worker acking multiple messages
It may be raising a duplicate ack error when ack was only called once per message
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.