RQ multiple workers with variable lifetimes. about flask-base

[abhisuri97]

http://python-rq.org/docs/results/ go to section on timeouts. flask-base currently doesn't support the functionality to set custom timeouts, but it seems simple enough to do.

[Mattpwinchester]

Thanks, that took care of timeouts for me. Now I just need to figure out how to get flask-base to have more than one redis queue worker active at a time. Is that something I can configure in the Local file or something I should have in manage.py?

[abhisuri97]

Yep. Just edit the Local file and I think you should be able to specify more workers (e.g. worker1, worker2 et al). Though I think this may be an anti pattern. Ideally, I think you would want to do something like listening for different types of worker queue calls. Ref http://python-rq.org/docs/workers/

[Mattpwinchester]

Do I have to specifiy the names for each worker? Can I say worker: python -u manage.py run_worker 100 to spin up 100 workers?

[abhisuri97]

Oh sorry, I should mention that though you can technically create 100+ workers...they all connect to the same port. You are probably going to need to do some more significant editing of the code in order to get multiple queues to work concurrently. I defer to @sandlerben for this since he implemented the worker code.

[Mattpwinchester]

Oh sorry, I should mention that though you can technically create 100+ workers...they all connect to the same port. You are probably going to need to do some more significant editing of the code in order to get multiple queues to work concurrently. I defer to @sandlerben for this since he implemented the worker code.

I tried spinning up 100 workers and I got the following error along with the server unable to start:

13:53:14 worker.1 | usage: manage.py [-?]
13:53:14 worker.1 | {reset,shell,setup_dev,format,rebuild,db,add_fake_data,runserver,setup_prod,test,recreate_db,run_worker,process_datasets}
13:53:14 worker.1 | ...
13:53:14 worker.1 | manage.py: error: too many arguments
13:53:14 worker.1 | exited with code 2

My Local file:

redis: redis-server
web: python -u manage.py runserver --host 127.0.0.1 --port 8001
worker: python -u manage.py run_worker 100

Please tell me what I need to change in order to have more than one worker currently running.

[sandlerben]

My experience with python-rq is actually fairly basic, so I'm not sure I'm well-equipped to answer this question. You will be best off asking this question on another forum, such as https://github.com/nvie/rq/issues or https://stackoverflow.com/questions/tagged/python-rq. Note that flask-rq, the library which flask-base uses to interface with python-rq, is just a slim wrapper around the standard python-rq library.