Increased base RAM usage and memory leak after upgrade from v23 to v24 in dockerized service about keycloak

[sschu]

@lnksz There are some background tasks running like the ones for expiring old session. The garbage collection can also be quite lazy to reclaim space. So my bet would be it is just not doing collections until it gets close to the maximum which is 70% of the 1G. Also I think the JVM doesnt necessarily return the memory to the OS but just keeps it. You could check by enabling metrics, there should you should see the number of small/large garbage collections and also how much heap space is actually used.

[mabartos]

@lnksz Thanks for the additional info. I just wanted to comment that it should be related only to the limit.

I created a follow-up issue to improve the docs: #28729

[shawkins]

Possibly related to #28211 and #28444

Did I miss any hint in https://www.keycloak.org/docs/latest/upgrading/index.html#migrating-to-24-0-0 on why this 3x RAM increase would be the case?

See #28211 (comment)

[lnksz]

@shawkins thanks for the quick reply! Sorry I didn't see #28211 because I only searched for open tickets...

With --memory=1G in the docker run command, the initial RAM usage of the the v24 container dropped to ca 500M, which is harmony with -XX:InitialRAMPercentage=50. I guess based on -XX:MaxRAMPercentage=70, this will go up to 700M, right?

What surprises me, that this RAM eating is taking place even though the KC container has 0 "load". No UI, requests, maintenance, anything...
Is it normal that KC continues to eat up the container memory in an idle state?

I will measure this now with the 1GB limit, because I would expect that an idle service find a (non max) heap size and stays there until some work needs to be done.

[shawkins]

this will go up to 700M, right?

Yes the heap will be allowed to grow up to 700M - the overall java process memory can be higher than that.

What surprises me, that this RAM eating is taking place even though the KC container has 0 "load". No UI, requests, maintenance, anything...
Is it normal that KC continues to eat up the container memory in an idle state?

I'm not sure what exactly is going on here. I would not expect the heap size to be adjusted higher unless there is pressure, and it doesn't seem like it would be done so smoothly / continuously. But it could be that the particular garbage collector in use will simply allow things to grow to MaxRAMPercentage over time based upon any amount of garbage (even just probes). I have not witnessed this behavior with local testing, so there may be some subtle difference running in Docker that needs another look. cc @mabartos @ahus1

[lnksz]

I have run the v24 as from the documentation with 1G limit. The RAM still grows slowly, even though I just let the service run without interacting with it in any way.

I captured the RAM usage via:

while true; do
	ram="$(docker stats --format=json --no-stream mykc | jq -r '.MemUsage' | cut -d' ' -f1)"
	ts=$(date +%s)
	echo "$ts,$ram" >> kc24-ram.csv
	sleep 30s
done

Then visualized it in Excel quickly.

[shawkins]

@lnksz and to confirm you never see out of memory execptions in your keycloak instance correct? As @sschu is mentions you can check the actual heap usage, and based upon other issues we don't expect that this memory growth actually means that keycloak is using more heap.

I recall @ahus1 mentioning seeing the tendency as well for the heap space to grow, but not be reduced.

If there are no actual out of memory problems, I'd vote for treating this issue as a normal priority to see if we should further influence the garbage collector to be more in-line with user expectations - it looks now like there's only a marginal benefit to setting a smaller initial ram percentage.

[lnksz]

Hi there,

I have run our "real" deployment of the KC configuration and monitored it over the weekend.
( with 1G limit in place)

So after the limit is set, over the long-run there is no such "run away" like with the unlimited version.

In the above screenshot one can see the v23 version (without limit) around 400M, then the unlimited v24 version (saw) and then the 1G limited v24 version.

This is now, what I would expect to see after an upgrade.
Maybe you could add a hint in the v24 upgrade guide? It is still not clear to me, what change exactly caused this in v24...