Django Channels Memory Leak on every message or connection about channels

[cacosandon]

Sure! I'll try to find time today to prepare a report on memray --leaks for each protocol server and test the PubSub layer. I'll get back to you soon, thanks!

[cacosandon]

So I tried multiple combinations. All HTML reports from memray are here:
reports.zip

But below there are screenshots from them.

First, tried with Redis Channels (not PubSub) to get memory leaks.

With uvicorn

PYTHONMALLOC=malloc memray run --force -o output.bin -m uvicorn core.asgi:application
+
memray flamegraph output.bin --force --leaks

So, the leaks report include memory that was never released back, but I don't know how to interpret it correctly. Seems like AuthMiddleware was leaking but after removing it, the results are almost the same.

redis-channels-uvicorn-leaks.html

Here is the screenshot of the uvicorn + leaks without AuthMiddleware:

redis-channels-uvicorn-without-authmiddleware-leaks.html

Then tried with daphne

PYTHONMALLOC=malloc memray run --force -o output.bin -m daphne core.asgi:application
+
memray flamegraph output.bin --force --leaks

redis-channels-daphne-leaks.html

The interesting part is that hypercorn showed no memory leaks (or maybe memray is not working here(?))

PYTHONMALLOC=malloc memray run --force -o output.bin -m hypercorn core.asgi:application
+
memray flamegraph output.bin --force --leaks

redis-channels-hypercorn-leaks.html

---

Then, I tried with garbage collect for uvicorn and daphne. Same story for both.

memray run --force -o output.bin -m uvicorn core.asgi:application
+
memray flamegraph output.bin --force

redis-channels-uvicorn-gccollect.html

memray run --force -o output.bin -m daphne core.asgi:application
+
memray flamegraph output.bin --force

redis-channels-daphne-gccollect.html

---

And finally tried with PubSub for uvicorn and daphne

memray run --force -o output.bin -m uvicorn core.asgi:application
+
memray flamegraph output.bin --force

redis-pubsub-uvicorn.html

memray run --force -o output.bin -m daphne core.asgi:application
+
memray flamegraph output.bin --force

redis-pubsub-daphne.html

---

Just in case, I also removed all @profile above functions so the memory leaks were not affected by the memory-profiler library.

Hope all these reports help understanding the constant memory increase.

Right now I am trying to move my application to hypercorn so I can test it on staging, but websocket messages are empty 🤔 . If I manage to solve it, I'll post the results here!

[cacosandon]

Hey @mitgr81! No updates yet.

We're using garbage collection on every message or new connection. This has helped a bit, but the memory still slowly increases and hits the max in about a week. We usually deploy and restart automatically the machines 2-3 times a week, which temporarily fixes the issue.

I hope the Channels team can look into this to see if it's a general problem with memory leaks. cc @carltongibson

[carltongibson]

Does the same thing happen with other protocol servers, such as hypercorn and Daphne?

[cacosandon]

I've tested daphne and hypercorn alongside uvicorn. All three show a similar pattern of memory usage, increasing steadily up to around 160 MiB. Despite this, they continue to consume more memory indefinitely, as monitored by memory-profiler.

The interesting thing is, while uvicorn shows a continuous rise in memory usage on the memray graph, the graphs for daphne and hypercorn are flat at 80 MiB. This discrepancy makes it unclear which tool provides more reliable data.

Here are the commands I used for each:

• Uvicorn: memray run --force -o output.bin -m uvicorn core.asgi:application
• Daphne: python -m memray run -o output.bin --force ./manage.py runserver
• Hypercorn: memray run --force -o output.bin -m hypercorn core.asgi:application

[carltongibson]

And can you see from any of the tools, memray perhaps, which objects are consuming the memory?

(I'd expect a gc.collect() to help here TBH)

[carltongibson]

@cacosandon Also, can you try with the PubSub layer, and see if the results are different there? Thanks.

[cacosandon]

I've made it to make hypercorn work!

For some reason the websocket messages that were bytes-only were sent as {"text": None, "bytes": ... } just in hypercorn so the function of AsyncWebsocketConsumer always called the text handler.

Added a PR for that: #2097

  async def websocket_receive(self, message):
      """
      Called when a WebSocket frame is received. Decodes it and passes it
      to receive().
      """
-     if "text" in message:
+     if "text" in message and message["text"] is not None:
          await self.receive(text_data=message["text"])
      else:
          await self.receive(bytes_data=message["bytes"])

Testing now in staging 🤞

[cacosandon]

Still there is a memory leak in my application with hypercorn 😕

It seems that memray just doesn't work with it, because memory-profiler does show a constant non-stop increase with any server protocol.

[carltongibson]

Hi @cacosandon

Looking at the uploaded report, for e.g. redis-pubsub-daphe, the memory usage rises and the stabilises:

The redis-channels-uvicorn-leaks report peaks at 168MB then falls to 151MB.

[cacosandon]

Hey @carltongibson, thank you for taking a look.

Yep, but if you zoom in redis-pubsub-daphe it just decelerates the memory increase (click on the graph). I think the first rise is just the correct memory usage, and then u see the memory leak.

On the other hand, redis-channels-uvicorn-leaks experiences memory drops at intervals due to the PYTHONMALLOC=malloc flag; however, the overall memory usage continues to increase. If you examine each drop, you'll notice that the memory level after each fall is higher than before, without stopping.

[cacosandon]

@carltongibson, do you have any clue about what's happening? Or what else can I try? I'm willing to try anything!

[carltongibson]

@cacosandon Given that you report it happening with the pub sub layer and different servers, not really. You need to identify where the leak is happening. Then it's possible to say something.

[cacosandon]

@carltongibson all my samples are from using RedisChannelLayer or RedisPubSubLayer, with uvicorn, daphne or hypercorn, with the tutorial example. My app has the problem too but I think it's a generalized problem.

Some things I've noticed:

• Memory increases constantly when there are large messages (>0.5MiB)
• Memory increases constantly when there are multiple connects/disconnects (every handshake adds memory)
• Memory leak is not present using InMemoryChannelLayer
• Using explicit del and gc.collect() decelerates the increase of memory.. but the leak is still present
• Creating large objects in Django Views does not leak the memory (every request kinds of clean up)

I don't know how nobody else is having this problem. Maybe they just don't send large messages 🤔

[carltongibson]

Hi @cacosandon — are you able to identify where the leak is happening? Alas, I haven't had time to dig into this further for you. Without that it's difficult to say too much.

If you can identify a more concrete issue, there's a good chance we can resolve it.

[cacosandon]

@carltongibson no :( that's actually the thing that I'm struggling on: finding the memory leak 😓

I really tried every tool to detect it, but nothing noticeable or strange in the reports..

[mitgr81]

I don't know how nobody else is having this problem. Maybe they just don't send large messages

I wouldn't assume that. 😉 I've been silently watching and hoping you find more than I did when I looked. We had some success changing servers from daphne to uvicorn. We're still seeing some leakiness, but have resolved to using tools to monitor memory and restart services.

Here are some other things I've watched:

• ansible/awx#7720
• #1181
• https://github.com/yuriymironov96/django-channels-leak (from django/daphne#373)

[cacosandon]

@mitgr81 what tools do you use to monitor and restart? For now I would love to implement that.

Will take a look on those resources!

[mitgr81]

@mitgr81 what tools do you use to monitor and restart? For now I would love to implement that.

We're rocking a bespoke monitor for docker containers. It's pretty simple; essentially we label each container with a valid restart time and a memory limit (among other rules); and the "container keeper" looks for them.

[mitgr81]

@cacosandon - Just curious if you've had any more luck than I have on this.