Celery integration API for channel about channels

[andrewgodwin]

Not going to happen; Celery and Channels provide very different running and transport guarantees (especially reliability).

[auvipy]

what If I want to use channels as celery tasks/worker?

[andrewgodwin]

Then someone would have to write a full compatability layer that included the celery Task API, retry logic, delay logic, periodic task scheduler, task acknowledgement, and a whole host of other features that Celery has that Channels doesn't, deliberately (adding them into the core would mean more code complexity and slower message processing)

[auvipy]

well, I saw honzakral work on something like this django/django#4763 is that a good starting point? thoughts?

[andrewgodwin]

That's a separate thing; any spec like that would be outside Channels and maybe a layer on top of it could implement it, but it's not in scope for this project.

[auvipy]

I am thinking to work on it separately so asking for your thoughts :D

[andrewgodwin]

Your main problem is that you'd have to use the database for a lot of state management; because messages in Channels aren't 100% guaranteed delivery, you'd need to store the sending of the task somewhere and have logic that updates a database when the job is started and when it's completed.

You also need to manually pipe task results into the database to deliver them, since Channels has no "task return value" mechanism; you only get to send a message to get processed, you don't get to wait for the result.

Thus, you'll have to do a lot of heavy lifting using the database as the "reliable" side to supplement Channels. You do get to avoid doing a queue in the database though and making things poll it for tasks, so that's a bonus.

Once there's a Tasks abstraction it might be worth taking a shot at one that fills it in based on channels; integrating with Celery might be a lot more work unless you drop a lot of the features there.

One thing Channels does probably need is some kind of periodic message injector kind of like celerybeat, but I'm not including that in initial plans as it's not essential to any of the core work, and limiting scope will help us ship.