quarkus-qe / quarkus-startstop Goto Github PK

Introduce COLD start mode for StartStopTest when the system caches are dropped before every Quarkus command execution

Details by Francesco Nigro:
Francesco Nigro: Hi!
Francesco Nigro: I'm currently running some experiment with hibernate + quarkus with a large JPA model
Francesco Nigro: I remember you have your own repo with the test run to check regressions
Francesco Nigro: for my tests I'm prepending this one

Francesco Nigro:
sync && sudo sh -c "echo 1 > /proc/sys/vm/drop_caches" && <quarkus app cmd>
that's making results much more predictable

Francesco Nigro: although obviously slower
Francesco Nigro: because will force the OS to release all OS page cache entries (including JVM shared libraries and class files previously read)
Francesco Nigro: but for your use case I think is better to be more deterministic then have faster results that won't happen in the real world
Francesco Nigro: (because in the real world you won't keep on restarting the same app over and over without doing anything more)

Rostislav Svoboda: caches drop can be applied before running the TS or it should be done before running every quarkus app command from the TS?

Francesco Nigro: before every each run of the quarkus app in theory
Francesco Nigro: because when you run it you populate the page cache with data
Francesco Nigro: and that will be used to speedup next time the startup
Francesco Nigro: but COLD startup requires everything to be uncached
Francesco Nigro: It could be another "mode" to run the same test i.e. let's call it "COLD" startup test
Francesco Nigro: I would keep it TBH, because is really much more realistic

Rostislav Svoboda: That cold startup mode would be Linux specific case with root/sudo. I would prefer either a specific switch or a new test.

Consider new scenario with Hibernate for StartStopTest,

ATM we cover jaxrs application + full micproprofile application.

Hibernate on the other hand brings in the need for DB and that can complicate the portability and our setup in perf lab.

The TS spits out various information, e.g. measurements, logs, maven commands and their parameters used etc.

When you are forming a report, you need to concatenate .csv files, correlate them with maven commands the TS wrote to log etc.

The TS should be able to form a "What I did" markdown report for each test, e.g. :

• maven command used
• measurements
• versions info

Failures in CI are related to new behavior in Quarkus 1.5.0.Final where not present in Jandex index issues are reported in dev mode too.

Reproducer mvn clean verify -Ptestsuite -DincludeTags=codequarkus

Move app-full-microprofile and app-jax-rs-minimal to RESTEasy Reactive with Quarkus 3 - https://groups.google.com/g/quarkus-dev/c/xjHIBk6Mx7k

This can happen once 2.13 branch is created.

Thresholds need to be evaluated and adjusted. RESTEasy Reactive has lower memory footprint and is faster.