OOMs and performance degradation after updating quarkus from 3.6.8 to 3.7.1 due to changed default jvm flags. about quarkus

[u6f6o]

@franz1981 I am no expert on this topic but this is one of the applications that ran into an OOM:

The jvm started with the desired committed heap (InitialRAMPercentage) but after a while it was shrinked. I found this on a redhat blog post:

Source: https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers

Are you mentioning OOM at java level or due to OS OOM killer?

OOM on java level

[geoand]

I'll leave it up to @franz1981 to decide

[quarkus-bot]

You added a link to a Zulip discussion, please make sure the description of the issue is comprehensive and doesn't require accessing Zulip

_{This message is automatically generated by a bot.}

[quarkus-bot]

/cc @geoand (jib,kotlin)

[geoand]

@jmtd I wonder if the choice for these flags needs to be revisited.

I'll cc @franz1981 in this discussion as well as he likely has plenty of knowledge to contribute

[u6f6o]

I forgot to share one thought in my initial post: According to my current knowledge, the jvm flags are put into a Properties object (thus a HashTable underneath) which means that the last flag provided wins (e.g. -XX:MaxRAMPercentage=80.0, -XX:MaxRAMPercentage=40.0 would lead to -XX:MaxRAMPercentage=40.0. (We currently try to reset the defaults by providing custom jib jvm flags).

I don't know though, which rules apply to"switchable" flags, like -XX:+ExitOnOutOfMemoryError,-XX:-ExitOnOutOfMemoryError. Does - or + win in such a case? Kind of the same situation if we want to use a different GC.

[franz1981]

We experienced these problems in the past as well (~1 year ago) and found out that these OOMs were caused by the resizing of the committed heap

Can you share more about this instead? G1 can return back heap memory (uncommitted) enabling periodic GCs, but I didn't hear anything about OOM cause by such. Are you mentioning OOM at java level or due to OS OOM killer?

[u6f6o]

We went for a workaround that resets the new jvm flags to their defaults:

The OOMs are gone now and our quarkus applications run smoothly again as they did before the quarkus update to 3.7 happened.

[geoand]

Thank you very much for the update!

[u6f6o]

@geoand : but why did you close it? Just because we found a workaround, this is still a bug imho? Up to my understanding, quarkus is supposed to run in different scenarios like typical every day joe webservers as well as heavy lifting batch jobs? Choosing a parallelGC and a very offensive heap shrinking behaviour as default is a very bad approach especially for the later one imo.