josiahcarlson / rpqueue Goto Github PK
View Code? Open in Web Editor NEWRedis Priority Queue offers a priority/timeline based queue for use with Redis
License: GNU Lesser General Public License v2.1
Redis Priority Queue offers a priority/timeline based queue for use with Redis
License: GNU Lesser General Public License v2.1
This change introduces a new bug:
6d9caa7#diff-2cee58761fd6da04a3218ebb31016238
To reproduce use @task decorator without parameters:
@rpqueue.task
def task1(a):
saw[0] = a
You will get error:
File "/opt/chownow/venv-hermosa/local/lib/python2.7/site-packages/rpqueue/__init__.py", line 1364, in execute_tasks
__import__(module) # for any connection modification side-effects
File "/opt/chownow/Hermosa/_tasks.py", line 7, in <module>
import tasks
File "/opt/chownow/Hermosa/tasks/__init__.py", line 7, in <module>
from . import billing
File "/opt/chownow/Hermosa/tasks/billing.py", line 10, in <module>
@close_session
File "/opt/chownow/venv-hermosa/local/lib/python2.7/site-packages/rpqueue/__init__.py", line 1233, in task
return decorate(args[0])
TypeError: 'function' object has no attribute '__getitem__'
*args is a list of parametrs and args is just one parameter
This code will throw exception: return decorate(args[0])
We have a priority task queue implementation in Nodejs https://github.com/LearnBoost/kue which uses a pattern like event notification sketch in http://redis.io/commands/blpop but thats not atomic and the helper list may be inconsistent with the actual tasks sorted set...
To overcome this I was thinking of using BLPOPRPUSH (to add reliability) and keep the list sorted as the main task container (atomicity) to provide priorities.
So I was thinking of using a background helper sorted set, using prioidical calls to SORT...STORE to keep the main list sorted. (but SORT...STORE doesn't trigger blocking clients)
until I faced your implementation which is very near to my mind. I think I can something like _handle_delayed_lua but I doubt I should use a small LIMIT, since large one can hurt new task priorities not being respected. (mine is not time-based, instead weighted tasks)
Line 918 in 7d099d1
Shouldn't this be setting the score to be new_t
rather than ARGV[1]
? It seems like the current behavior would just immediately expire every task instead.
Add the ability to automatically retry a task if it has taken longer than a specified number of seconds to execute within the task processor.
Ref : https://www.python.org/dev/peps/pep-0632/#migration-advice
setup from setuptools can be used.
Line 3 in 15f1946
Where is the documentation around the integer passed to save_results, maybe
https://github.com/josiahcarlson/rpqueue/blame/master/README.rst#L161
We removed certain cron_tasks and restarted our rpqueue server. The tasks remained scheduled and when their time came, a KeyError arose in _get_work()
during the following if
branch:
if item_id == work[1]:
# periodic or cron task, reschedule it
sch = work[-1] if item_id in NEVER_SKIP else time.time()
delay = REGISTRY[item_id].next() # KeyError here
All of our worker threads died as a result of this KeyError.
In principle this can be avoided, but we had no way to predict it nor reason to expect it. It seems to me a warning rather than a full exception is a more appropriate response. Other parts of rpqueue
server code do check for the function name before acting on it, so it makes sense to do so here as well.
PR coming shortly.
We spent an awful long time debugging insane defaults and a printout akin to " scheduled to execute every 9999999 seconds" would have helped.
Is it possible that the rpqueue.run method or the whole project is not compatible with Python3? I wanted to start the execution daemon and got the following error message:
File "/virtenv/lib/python3.6/site-packages/rpqueue/__init__.py", line 881, in execute_tasks
for p in xrange(processes):
NameError: name 'xrange' is not defined
After switching to python2 the executor started as expected.
If so you should probably mention it somewhere in the documentation. Otherwise I would love to know why it does not work for me.
Thanks!
Add the ability to put a task in a different queue. Can be used for re-prioritization, etc.
Thanks
No, seriously, I should be ashamed for the awful logging configuration that rpqueue uses by default. I need to fix that.
When a task finishes, it may return something.
Offer the ability to store the result into a tombstone, which can then be referenced by the caller.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.