frkhit / celery-redis-priority-test

Used to determine how priorities are working in celery with a redis backend

celery-redis-priority-test

Used to determine how priorities are working in celery with a redis backend

run in docker

docker-compose up

How to do the thing:

run celery in one shell celery worker -A tasks -Q a-high,b-medium,c-low -Ofair -c1 --prefetch-multiplier=1

and run tests in another green (i dont have pytest commented yet. will do soon)

https://medium.com/@hengfeng/how-to-route-tasks-in-celery-rabbitmq-e0e22c178e45

Undocumented, or incorrect assertions of Celery and Redis:

1: task_inherit_parent_priority is available for the Redis backend, and not just AMQP

2: Task priority is low to high, eg: tasks with a priority of 0 are handeled before 9

3: If you set the broker_transport_options to {"queue_order_strategy": "sorted"}, it will prioritize tasks from queues in alphabetical order

4: Tasks are ran in order of sorting first by task priority, and second by queue order

5: If you override task priority by setting broker_transport_options to {"priority_steps":[0, 2, 4, 6, 8]}, 0 will be the highest priority. If you invert that to {"priority_steps":[8, 6, 4, 2, 0]}, 8 will be the highest. The order of priority_steps dictates priority order.