Enable automatic reload upon source code changes about rq

[honi]

This is how I'm currently autoreloading the worker process. This is a management command I called devrqworker, so instead of running manage.py rqworker I simply run manage.py devrqworker during dev.

import os
import shlex
import subprocess

from django.core.management.base import BaseCommand
from django.utils.autoreload import run_with_reloader


class Command(BaseCommand):
    def add_arguments(self, parser):
        parser.add_argument('queues', nargs='+', type=str)
        parser.add_argument('--worker-pid-file', action='store', dest='worker_pid_file', default='/tmp/rqworker.pid')

    def handle(self, *args, **options):
        run_with_reloader(run_worker, options['queues'], options['worker_pid_file'])


def run_worker(queues, worker_pid_file):
    if os.path.exists(worker_pid_file):
        worker_pid = subprocess.run(['cat', worker_pid_file], stdout=subprocess.PIPE).stdout.decode('utf-8')
        kill_worker_cmd = f'kill {worker_pid}'
        subprocess.run(shlex.split(kill_worker_cmd), stderr=subprocess.PIPE)

    queues = ' '.join(queues)
    start_worker_cmd = f'{get_managepy_path()} rqworker {queues} --pid={worker_pid_file}'
    subprocess.run(shlex.split(start_worker_cmd))


def get_managepy_path():
    managepy_path = None
    search_path = os.path.dirname(__file__)
    while not managepy_path:
        if os.path.exists(os.path.join(search_path, 'manage.py')):
            managepy_path = os.path.abspath(os.path.join(search_path, 'manage.py'))
        else:
            search_path = os.path.join(search_path, '../')
    return managepy_path

[ffigiel]

One solution for this would be to write a script that runs workers in --burst mode infinitely.

Example:

#!/usr/bin/env bash
while true; do
  rqworker --burst
  sleep 1
done

Of course it won't work reliably if you have long monolithic jobs, but in that case you have a bigger problem than reloading your workers ;)

[theomessin]

Was going through the code and I couldn't really see why you'd have to reload. Aren't functions "imported" using getattr during execution:

rq/rq/job.py

Lines 191 to 200 in 549648b

	@property
	def func(self):
	func_name = self.func_name
	if func_name is None:
	return None
	if self.instance:
	return getattr(self.instance, func_name)
	return import_attribute(self.func_name)

I tested this real-quick by starting a worker where the job initially made a file "old" and issued a job (the file "old" was made). Then I deleted that file, changed the filename to "new", and issued another job (without restarting the worker). A "new" file was made. Am I missing something?

[sidneijp]

There's a package called django-rq-wrapper that implements autoreload and start multiple workers. I didn't test it, but seems pretty simple and not evasive since it brings a new management command called "rqworkers".

https://github.com/istrategylabs/django-rq-wrapper

[honi]

This would be a nice feature indeed +1

[mschettler]

i support this

[honi]

Can you manually send a signal to reload a worker?

[theomessin]

Any update on this?

[selwin]

No one is working on this as far as I know. It would be nice to have this feature though. Django uses watchman to trigger reloads. I'd accept a PR that implements live reload functionality :)

[theomessin]

Thanks for the update. I'll see if I find the time to submit a PR. For now just went with using entr with ack (doesn't work if you add new files though):

ack -f | entr -r -s "rq worker -u redis://redis:6379"

[taewookim]

@honi THanks. Works well.

PSA: Don't use it in production. This eats HELLA memory. Like 100 mb (regular rq worker) vs 300 mb (this custom version)

[honi]

Correct! In production you should use the default rqworker management command.

[rpkak]

Like in #2 (comment) I made a simple test and it worked. Can someone send me an example so I can look at it.

[ccrvlh]

When you enqueue() something, it creates a job using the reference for the function/callable (which can be both a string or a callable, if the latter, it will be converted to a string - basically the import path). This sets a couple of private attributes to the job (_func_name and _instance), those attributes will then be used by the job itself to getattr or import the relevant code (this happens on the func property of the job).

What the worker does is basically just call the perform method of the job, that's what actually executes the callable. So indeed, there shouldn't be a need to add a reload functionality, since the worker doesn't know the job before it actually pulls it from the queue.

If anyone has any issues with this, feel free to open a new issue so we can investigate. Closing it for now.

[aksu-104128]

Python will cache modules that have already been imported, and the default worker will fork new processes when executing tasks, so it will not be affected by module caching. If you use simple worker, there will be problems when making code changes