quarkus-test-suite's People
Forkers
sgitario pjgg mjurc jsmrcka rsvoboda fedinskiy kshpak penehyba dale-fu davided alexrogalskiy gsmet jcarranzan michalvavrik ppalaga marcnuri-forks afalhambra iamvcm848 gtroitsk linartova mikeedgar simranpal jedla97 mocenas tsegismont valen-mascarenhas14 dipeshsingh253 brunobat ramdrquarkus-test-suite's Issues
Upgrade Quarkus to 2.x
README mentions ephemeral namespaces but there is no such section
https://github.com/quarkus-qe/quarkus-test-suite#running-against-red-hat-build-of-quarkus mentions ephemeral namespaces but there is no such section in the README
If you want to run the tests using ephemeral namespaces, go to the [Running tests in ephemeral namespaces](#running-tests-in-ephemeral-namespaces) section for more information.
Enhance readme to add more info about the framework and how to run native mode / against OpenShift, Kubernetes
Enhance readme of the TS to add more info about the framework and how to run native mode
https://github.com/quarkus-qe/quarkus-test-framework#readme
https://github.com/quarkus-qe/quarkus-test-framework#native
The details need to be at the top
Current one-liner (https://github.com/quarkus-qe/quarkus-test-suite#test-framework) is not sufficiently visible and strong on the message.
Add details how to run against OpenShift, Kubernetes.
Migrate 602-spring-data-rest from the Quarkus Beefy Test Suite
Warning about path to the parent when running the tests
When running http-minimal
module I noticed warning about path to the parent
15:17:21.961 INFO Running command: mvn install -Dcheckstyle.skip -DskipTests=true -DskipITs=true -pl .
...
15:17:29.448 INFO [app] [WARNING] 'parent.relativePath' of POM io.quarkus.ts.qe:http-minimum:1.0.0-SNAPSHOT
(/Users/rsvoboda/git/quarkus-test-suite/http/http-minimum/target/app/pom.xml)
points at io.quarkus.ts.qe:http-minimum instead of io.quarkus.ts.qe:parent,
please verify your project structure @ line 4, column 13
The warning is imho because the app is copied into target directory and from there it points two levels up, should be pointing 4 levels up to behave correctly.
I wonder if the <relativePath>../..</relativePath>
is really needed.
Migrate 603-spring-web-smallrye-openapi from the Quarkus Beefy Test Suite
Add CI to run tests on Windows for PRs and Daily actions
Depends on quarkus-qe/quarkus-test-framework#141
Migrate 009-quarkus-infinispan-grpc-kafka from the Quarkus Beefy Test Suite
Migrate 010-quarkus-opentracing-reactive-grpc from the Quarkus Beefy Test Suite
Migrate 023-quarkus-kamelet from the Quarkus Beefy Test Suite
Docs: Effective way to investigate test failures
When working on migration of "many extensions" scenario I had to investigate failure with the latest Quarkus main, Quarkus 2.1.0.Final was working fine.
In BeefyTS I can run the module in dev mode (as it is vanilla Quarkus app) and experiment , but in this TS I had much longer turnaround cycle, in the end I had to add Thread.sleep
into the test to be able to play a bit more with the application.
When I realized the problem is in some extension or combination of extensions I was struggling for the second time. Adding / Removing extensions in vanilla Quarkus app with dev mode is so seamless, just edit the pom.xml, refresh browser window. With this TS it's more cumbersome, no dev mode, longer cycle to get feedback. I ended up with standalone Quarkus app and I copied the files from the module and started experimenting with dev mode.
Experience with investigation using this TS is worse than with BeefyTS.
We should start thinking about more effective ways to investigate test failures, how to prepare the standalone app.
I really miss the dev mode ability.
Migrate 015-quarkus-micrometer into micrometer/prometheus from the Quarkus Beefy Test Suite
Migrate 002-quarkus-all-extensions from the Quarkus Beefy Test Suite
Migrate 601-spring-data-primitive-types from the Quarkus Beefy Test Suite
Both modules should be one in the new test suite.
Migrate 006-quartz-manually-scheduled-job and 017-quartz-cluster from the Quarkus Beefy Test Suite
These modules should be migrated into a single module.
Add coverage for `quarkus-logging-json`
Related to #157
Migrate External Applications
Split CI native runs into more parallel runs
Split CI native runs into more parallel runs
Waiting 4 hours for PR run is too much (#58)
Migrate 101-javaee-like-getting-started from the Quarkus Beefy Test Suite
Upgrade AMQ Streams to 1.7.0
In order to be aligned to Quarkus upstream: quarkusio/quarkus#14732
Similarly to https://github.com/quarkus-qe/quarkus-openshift-test-suite/pull/279/files
Migrate external applications from the Quarkus OpenShift Test Suite
These three modules need to be migrated to the existing test suite: https://github.com/quarkus-qe/quarkus-openshift-test-suite/tree/main/external-applications
Migrate 022-quarkus-properties-config-all from the Quarkus Beefy Test Suite
Add servlet undertow scenario + session eviction
Add quarkus-undertow
extension coverage, focus on Http session eviction
Migrate 303-quarkus-vertx-sql from the Quarkus Beefy Test Suite
Under sql-db subfolder
Add coverage for `quarkus-cache`
Related to #157
Migrate the Infinispan Client module from the Quarkus OpenShift Test Suite
The infinispan-client module needs to be migrated to this test suite: https://github.com/quarkus-qe/quarkus-openshift-test-suite/tree/main/infinispan-client
Migrate 005-quarkus-and-maven-profiles from the Quarkus Beefy Test Suite
Migrate SSL to Kstreams and reactive Kafka coverage into messaging/kafka-streams-reactive-messaging
Changes to be migrated: quarkus-qe/beefy-scenarios#237
Tests for quarkus-cli fail
On current main (commit da63a00 at the time of writing) all tests in quarkus-cli module fail with exception Caused by: java.io.IOException: Cannot run program "quarkus" (in directory "target"): error=2, No such file or directory
.
Migrate 018-quarkus-runtime-properties from the Quarkus Beefy Test Suite
Migrate 003-quarkus-many-extensions from the Quarkus Beefy Test Suite
Migrate 304-quarkus-vertx-routes from the Quarkus Beefy Test Suite
Under http subfolder
Docs: Unable to run just one test case using `-Dtest=ABC`
Unable to run just one test case using -Dtest=ABC
I can run the test using -Dit.test=ABC
, but I think people are used to do just -Dtest=ABC
aka surefire way.
@Sgitario, I was wondering if TS / failsafe plugin can give some warning to remind people to use -Dit.test=ABC
instead of-Dtest=ABC
?
Command examples:
mvn clean verify -pl http/http-minimum -Dtest=io.quarkus.ts.http.minimum.HttpMinimumIT
vs.
mvn clean verify -pl http/http-minimum -Dit.test=io.quarkus.ts.http.minimum.HttpMinimumIT
Migrate 011-quarkus-panache-rest-flyway and 014-quarkus-panache-with-transactions-xa from the Quarkus Beefy Test Suite
The new module should be part of sql-db parent and both modules should be one in the new test suite.
Migrate 001-quarkus-getting-started-with-jaxrs from the Quarkus Beefy Test Suite
I would add it into the http/http-minimum or http/http-advanced modules. Wdyt @rsvoboda ?
Testsuite gets stuck for 5 minutes when the tested application misbehaves
Testsuite gets stuck when the tested application misbehaves like in case of https://github.com/rsvoboda/quarkus-test-suite/tree/many-extensions
git clone https://github.com/rsvoboda/quarkus-test-suite
cd quarkus-test-suite
git checkout many-extensions
mvn clean verify -pl super-size/many-extensions
Just copied stuff from http-minimal and added extensions from https://github.com/quarkus-qe/beefy-scenarios/blob/main/003-quarkus-many-extensions/pom.xml. Intentionally didn't add content from application.properties.
I think there should be quicker fail for "baremetal" applications by default.
I know one can tune timeouts, but the default should be more fail fast friendly for the "baremetal" env.
What I see is:
[INFO] Running io.quarkus.ts.http.minimum.DevModeHttpMinimumIT
15:26:18.836 INFO [app] Initialize service
...
15:26:43.038 INFO [app] 2021-07-23 15:26:40,320 INFO [org.jbo.threads] (main) JBoss Threads version 3.4.0.Final
...
15:36:19.468 WARNING [app] Service didn't start in PT5M minutes
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 300.218 s <<< FAILURE! - in io.quarkus.ts.http.minimum.HttpMinimumIT
Investigate approach about how to run tests that requires high execution times
An example of a scenario that requires a lot of execution time to run is: https://github.com/quarkus-qe/quarkus-test-suite/blob/main/scheduling/quartz/src/test/java/io/quarkus/qe/scheduling/quartz/AnnotationScheduledJobsMySqlQuartzIT.java
When running this test on Native, it will build the Native app three times (because we're selecting the classes to include).
Therefore, we need to find an approach to run these scenarios.
Missing coverage of a few RHBQ supported extensions
We have few extensions that miss coverage in this TS.
- quarkus-cache: #165
- quarkus-logging-json: #168
- quarkus-rest-client-jaxb: #170
- quarkus-rest-client-jsonb: #170
- quarkus-spring-boot-properties: #172
- quarkus-spring-cache: #179
- quarkus-spring-cloud-config-client: #184
- quarkus-spring-scheduled: #186
Used commands:
mvn dependency:list -Dinclude.serverless -Dinclude.operator-scenarios -Dinclude.quarkus-cli-tests | grep ':jar:' | sort | uniq | grep io.quarkus | sed "s/.*io.quarkus/io.quarkus/g" | cut -d: -f 1-2 | uniq | grep -v io.quarkus.ts > ~/Downloads/quarkus-test-suite-extensions.txt
for i in `cat ~/Downloads/RHBQ-supported.txt`; do echo "$i ... `grep $i ~/Downloads/quarkus-test-suite-extensions.txt | wc -l`"; done | grep 0 | sort
for i in `cat ~/Downloads/RHBQ-supported.txt`; do echo "$i ... `grep $i ~/Downloads/quarkus-test-suite-extensions.txt | wc -l`"; done | grep 0 | wc -l
Update README
- Developers need starters for development and experiments.
- Point out to the developer the location of examples and framework readme files.
Upgrade Kafka to use 2.7
This commit quarkus-qe/quarkus-openshift-test-suite@5d9eadd needs to be merged into the test suite.
Basically, we need to update this property:
Line 290 in 917e031
For the strimzi container, we're using the image 0.22.1-kafka-2.7.0
(this comes from the test framework - https://github.com/quarkus-qe/quarkus-test-framework/blob/1b1736cd92c9b57716461b06e97f7766512ef446/quarkus-test-service-kafka/src/main/java/io/quarkus/test/services/containers/model/KafkaVendor.java#L5), but it can be replaced using the Kafka container annotation.
Migrate 004-quarkus-HHH-and-HV from the Quarkus Beefy Test Suite
Migrate quarkus-oidc-token-propagation coverage into security/keycloak-oidc-client
Migrate 201-large-static-content from the Quarkus Beefy Test Suite
Maybe this should be part of the http/http-minimum or advanced? wdty @rsvoboda ?
Migrate 302-quarkus-vertx-jwt from the Quarkus Beefy Test Suite
Under security subfolder
Add `-e` switch to maven commands executed by the TS, can be handy in dev mode tests
Add -e
switch to maven commands executed by the TS, can be handy in dev mode tests
When I was working on migration of "many extensions" module I noticed failed run of the dev mode tests, apparently something in quarkus maven plugin, but no details about that.
Running commands with -e
would help with the investigation.
17:25:42.192 INFO [app] 17:25:39,788 Profile DevModeManyExtensionsIT activated. Live Coding activated.
17:25:42.192 INFO [app] 17:25:39,788 Installed features: [agroal, artemis-jms, cdi, config-yaml, hibernate-orm, hibernate-orm-panache, hibernate-validator, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-postgresql, kafka-client, keycloak-authorization, kubernetes, narayana-jta, oidc, quartz, reactive-pg-client, rest-client, resteasy, resteasy-jackson, resteasy-jaxb, resteasy-jsonb, scheduler, security, servlet, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-jwt, smallrye-metrics, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-amqp, smallrye-reactive-messaging-kafka, spring-data-jpa, spring-di, spring-security, spring-web, swagger-ui, vertx, vertx-web, websockets, websockets-client]
17:25:42.192 INFO [app] [INFO] ------------------------------------------------------------------------
17:25:42.193 INFO [app] [INFO] BUILD FAILURE
17:25:42.193 INFO [app] [INFO] ------------------------------------------------------------------------
17:25:42.193 INFO [app] [INFO] Total time: 22.841 s
17:25:42.193 INFO [app] [INFO] Finished at: 2021-08-02T17:25:41+02:00
17:25:42.193 INFO [app] [INFO] ------------------------------------------------------------------------
17:25:42.193 INFO [app] [ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:2.1.0.Final:dev (default-cli) on project many-extensions: Failed to run: Dev mode process did not complete successfully -> [Help 1]
17:25:42.193 INFO [app] [ERROR]
17:25:42.193 INFO [app] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
17:25:42.193 INFO [app] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
17:25:42.193 INFO [app] [ERROR]
17:25:42.194 INFO [app] [ERROR] For more information about the errors and possible solutions, please read the following articles:
17:25:42.194 INFO [app] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
17:25:43.675 INFO [app] Service stopped
17:25:44.748 INFO [app] Service stopped
[INFO] Tests run: 60, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.751 s - in io.quarkus.ts.many.extensions.DevModeManyExtensionsIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 120, Failures: 0, Errors: 0, Skipped: 0
Migrate 301-quarkus-vertx-kafka from the Quarkus Beefy Test Suite
@pjgg I see the coverage is a bit similar to what we have in messaging/kafka-avro-reactive-messaging but without vertex.
Pls, analyse if the coverage here can be added in this module or create a new module under messaging/.
Migrate 300-quarkus-vertx-webclient from the Quarkus Beefy Test Suite
As part of the http subfolder.
Build of the TS takes 01:37 min without tests, should be faster for just 28 modules
mvn clean package -DskipTests
takes 01:37 min on my MBP, it's third run to avoid spending time downloading.
There are few things to look into:
- try to identify why it takes so long
- try to optimize current situation
- implement monitoring on size / build time characteristics / tests execution time / health of the testsuite
- we have Jaeger tracing service, we should have daily jobs to push data to it and some (Grafana?) reports around that
- new TS is crutial for us and we need to keep monitoring it
- regularly evaluate coverage in the TS, structure of the modules, packages / TestCase / Test names
mvn clean package -DskipTests
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Quarkus QE TS: Parent 1.0.0-SNAPSHOT:
[INFO]
[INFO] Quarkus QE TS: Parent .............................. SUCCESS [ 2.498 s]
[INFO] Quarkus QE TS: Config .............................. SUCCESS [ 6.126 s]
[INFO] Quarkus QE TS: http-minimum ........................ SUCCESS [ 1.814 s]
[INFO] Quarkus QE TS: http-advanced ....................... SUCCESS [ 6.900 s]
[INFO] Quarkus QE TS: Scaling tests ....................... SUCCESS [ 2.147 s]
[INFO] Quarkus QE TS: Micrometer: Prometheus .............. SUCCESS [ 4.323 s]
[INFO] Quarkus QE TS: Micrometer: Prometheus with Kafka ... SUCCESS [ 7.369 s]
[INFO] Quarkus QE TS: Micrometer: OIDC .................... SUCCESS [ 4.538 s]
[INFO] Quarkus QE TS: Messaging: Artemis .................. SUCCESS [ 2.528 s]
[INFO] Quarkus QE TS: Messaging: Artemis + JTA ............ SUCCESS [ 2.067 s]
[INFO] Quarkus QE TS: Messaging: AMQP with Reactive Messaging SUCCESS [ 2.898 s]
[INFO] Quarkus QE TS: Messaging: QPID ..................... SUCCESS [ 2.052 s]
[INFO] Quarkus QE TS: Messaging: Reactive Kafka ........... SUCCESS [ 2.936 s]
[INFO] Quarkus QE TS: Messaging: Reactive Kafka with Avro . SUCCESS [ 3.819 s]
[INFO] Quarkus QE TS: MicroProfile ........................ SUCCESS [ 2.097 s]
[INFO] Quarkus QE TS: Security: Basic ..................... SUCCESS [ 2.154 s]
[INFO] Quarkus QE TS: Security: HTTPS ..................... SUCCESS [ 2.352 s]
[INFO] Quarkus QE TS: Security: JWT ....................... SUCCESS [ 2.678 s]
[INFO] Quarkus QE TS: Security: Keycloak .................. SUCCESS [ 2.410 s]
[INFO] Quarkus QE TS: Security: Keycloak + Authorization .. SUCCESS [ 3.380 s]
[INFO] Quarkus QE TS: Security: Keycloak + JWT ............ SUCCESS [ 3.240 s]
[INFO] Quarkus QE TS: Security: Keycloak + Webapp ......... SUCCESS [ 5.027 s]
[INFO] Quarkus QE TS: Security: Keycloak + OAuth2 ......... SUCCESS [ 5.035 s]
[INFO] Quarkus QE TS: Security: Keycloak + MultiTenant .... SUCCESS [ 4.967 s]
[INFO] Quarkus QE TS: Security: Keycloak + OIDC Client .... SUCCESS [ 3.722 s]
[INFO] Quarkus QE TS: SQL Database: Application ........... SUCCESS [ 3.548 s]
[INFO] Quarkus QE TS: SQL Database: Multiple Persistence Units SUCCESS [ 3.275 s]
[INFO] Quarkus QE TS: Quarkus CLI ......................... SUCCESS [ 0.861 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:37 min
[INFO] Finished at: 2021-07-21T15:20:57+02:00
[INFO] ------------------------------------------------------------------------
Migrate 019-quarkus-consul from the Quarkus Beefy Test Suite
The new module should be part of config parent
Migrate the Lifecycle application module from the Quarkus OpenShift Test Suite
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.