Coder Social home page Coder Social logo

quarkus-test-suite's People

Contributors

afalhambra avatar dale-fu avatar dependabot[bot] avatar dipeshsingh253 avatar fedinskiy avatar gsmet avatar gtroitsk avatar jcarranzan avatar jedla97 avatar jsmrcka avatar kshpak avatar linartova avatar michalvavrik avatar mikeedgar avatar mjurc avatar mocenas avatar ramdr avatar rsvoboda avatar sgitario avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

quarkus-test-suite's Issues

Enhance readme to add more info about the framework and how to run native mode / against OpenShift, Kubernetes

Enhance readme of the TS to add more info about the framework and how to run native mode

https://github.com/quarkus-qe/quarkus-test-framework#readme
https://github.com/quarkus-qe/quarkus-test-framework#native

The details need to be at the top

Current one-liner (https://github.com/quarkus-qe/quarkus-test-suite#test-framework) is not sufficiently visible and strong on the message.

Add details how to run against OpenShift, Kubernetes.

Warning about path to the parent when running the tests

When running http-minimal module I noticed warning about path to the parent

15:17:21.961 INFO  Running command: mvn install -Dcheckstyle.skip -DskipTests=true -DskipITs=true -pl .
... 
15:17:29.448 INFO  [app] [WARNING] 'parent.relativePath' of POM io.quarkus.ts.qe:http-minimum:1.0.0-SNAPSHOT 
  (/Users/rsvoboda/git/quarkus-test-suite/http/http-minimum/target/app/pom.xml) 
  points at io.quarkus.ts.qe:http-minimum instead of io.quarkus.ts.qe:parent, 
  please verify your project structure @ line 4, column 13

The warning is imho because the app is copied into target directory and from there it points two levels up, should be pointing 4 levels up to behave correctly.

I wonder if the <relativePath>../..</relativePath> is really needed.

Docs: Effective way to investigate test failures

When working on migration of "many extensions" scenario I had to investigate failure with the latest Quarkus main, Quarkus 2.1.0.Final was working fine.

In BeefyTS I can run the module in dev mode (as it is vanilla Quarkus app) and experiment , but in this TS I had much longer turnaround cycle, in the end I had to add Thread.sleep into the test to be able to play a bit more with the application.

When I realized the problem is in some extension or combination of extensions I was struggling for the second time. Adding / Removing extensions in vanilla Quarkus app with dev mode is so seamless, just edit the pom.xml, refresh browser window. With this TS it's more cumbersome, no dev mode, longer cycle to get feedback. I ended up with standalone Quarkus app and I copied the files from the module and started experimenting with dev mode.

Experience with investigation using this TS is worse than with BeefyTS.

We should start thinking about more effective ways to investigate test failures, how to prepare the standalone app.
I really miss the dev mode ability.

Tests for quarkus-cli fail

On current main (commit da63a00 at the time of writing) all tests in quarkus-cli module fail with exception Caused by: java.io.IOException: Cannot run program "quarkus" (in directory "target"): error=2, No such file or directory.

Docs: Unable to run just one test case using `-Dtest=ABC`

Unable to run just one test case using -Dtest=ABC

I can run the test using -Dit.test=ABC, but I think people are used to do just -Dtest=ABC aka surefire way.

@Sgitario, I was wondering if TS / failsafe plugin can give some warning to remind people to use -Dit.test=ABC instead of-Dtest=ABC?

Command examples:

mvn clean verify  -pl http/http-minimum -Dtest=io.quarkus.ts.http.minimum.HttpMinimumIT
  vs.
mvn clean verify  -pl http/http-minimum -Dit.test=io.quarkus.ts.http.minimum.HttpMinimumIT

Testsuite gets stuck for 5 minutes when the tested application misbehaves

Testsuite gets stuck when the tested application misbehaves like in case of https://github.com/rsvoboda/quarkus-test-suite/tree/many-extensions

git clone https://github.com/rsvoboda/quarkus-test-suite
cd quarkus-test-suite
git checkout many-extensions
mvn clean verify  -pl super-size/many-extensions

Just copied stuff from http-minimal and added extensions from https://github.com/quarkus-qe/beefy-scenarios/blob/main/003-quarkus-many-extensions/pom.xml. Intentionally didn't add content from application.properties.

I think there should be quicker fail for "baremetal" applications by default.
I know one can tune timeouts, but the default should be more fail fast friendly for the "baremetal" env.

What I see is:

[INFO] Running io.quarkus.ts.http.minimum.DevModeHttpMinimumIT
15:26:18.836 INFO  [app] Initialize service
...
15:26:43.038 INFO  [app] 2021-07-23 15:26:40,320 INFO  [org.jbo.threads] (main) JBoss Threads version 3.4.0.Final
...
15:36:19.468 WARNING [app] Service didn't start in PT5M minutes
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 300.218 s <<< FAILURE! - in io.quarkus.ts.http.minimum.HttpMinimumIT

Investigate approach about how to run tests that requires high execution times

An example of a scenario that requires a lot of execution time to run is: https://github.com/quarkus-qe/quarkus-test-suite/blob/main/scheduling/quartz/src/test/java/io/quarkus/qe/scheduling/quartz/AnnotationScheduledJobsMySqlQuartzIT.java

When running this test on Native, it will build the Native app three times (because we're selecting the classes to include).

Therefore, we need to find an approach to run these scenarios.

Missing coverage of a few RHBQ supported extensions

We have few extensions that miss coverage in this TS.

  • quarkus-cache: #165
  • quarkus-logging-json: #168
  • quarkus-rest-client-jaxb: #170
  • quarkus-rest-client-jsonb: #170
  • quarkus-spring-boot-properties: #172
  • quarkus-spring-cache: #179
  • quarkus-spring-cloud-config-client: #184
  • quarkus-spring-scheduled: #186

Used commands:

mvn dependency:list -Dinclude.serverless -Dinclude.operator-scenarios -Dinclude.quarkus-cli-tests | grep ':jar:' | sort | uniq | grep io.quarkus | sed "s/.*io.quarkus/io.quarkus/g" | cut -d: -f 1-2 | uniq | grep -v io.quarkus.ts > ~/Downloads/quarkus-test-suite-extensions.txt
for i in `cat ~/Downloads/RHBQ-supported.txt`; do echo "$i ... `grep $i ~/Downloads/quarkus-test-suite-extensions.txt | wc -l`"; done | grep 0 | sort
for i in `cat ~/Downloads/RHBQ-supported.txt`; do echo "$i ... `grep $i ~/Downloads/quarkus-test-suite-extensions.txt | wc -l`"; done | grep 0 | wc -l

Update README

  • Developers need starters for development and experiments.
  • Point out to the developer the location of examples and framework readme files.

Upgrade Kafka to use 2.7

This commit quarkus-qe/quarkus-openshift-test-suite@5d9eadd needs to be merged into the test suite.
Basically, we need to update this property:

registry.redhat.io/amq7/amq-streams-kafka-24-rhel7

For the strimzi container, we're using the image 0.22.1-kafka-2.7.0 (this comes from the test framework - https://github.com/quarkus-qe/quarkus-test-framework/blob/1b1736cd92c9b57716461b06e97f7766512ef446/quarkus-test-service-kafka/src/main/java/io/quarkus/test/services/containers/model/KafkaVendor.java#L5), but it can be replaced using the Kafka container annotation.

Add `-e` switch to maven commands executed by the TS, can be handy in dev mode tests

Add -e switch to maven commands executed by the TS, can be handy in dev mode tests

When I was working on migration of "many extensions" module I noticed failed run of the dev mode tests, apparently something in quarkus maven plugin, but no details about that.

Running commands with -e would help with the investigation.

    17:25:42.192 INFO  [app] 17:25:39,788 Profile DevModeManyExtensionsIT activated. Live Coding activated.
    17:25:42.192 INFO  [app] 17:25:39,788 Installed features: [agroal, artemis-jms, cdi, config-yaml, hibernate-orm, hibernate-orm-panache, hibernate-validator, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-postgresql, kafka-client, keycloak-authorization, kubernetes, narayana-jta, oidc, quartz, reactive-pg-client, rest-client, resteasy, resteasy-jackson, resteasy-jaxb, resteasy-jsonb, scheduler, security, servlet, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-jwt, smallrye-metrics, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-amqp, smallrye-reactive-messaging-kafka, spring-data-jpa, spring-di, spring-security, spring-web, swagger-ui, vertx, vertx-web, websockets, websockets-client]
    17:25:42.192 INFO  [app] [INFO] ------------------------------------------------------------------------
    17:25:42.193 INFO  [app] [INFO] BUILD FAILURE
    17:25:42.193 INFO  [app] [INFO] ------------------------------------------------------------------------
    17:25:42.193 INFO  [app] [INFO] Total time:  22.841 s
    17:25:42.193 INFO  [app] [INFO] Finished at: 2021-08-02T17:25:41+02:00
    17:25:42.193 INFO  [app] [INFO] ------------------------------------------------------------------------
    17:25:42.193 INFO  [app] [ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:2.1.0.Final:dev (default-cli) on project many-extensions: Failed to run: Dev mode process did not complete successfully -> [Help 1]
    17:25:42.193 INFO  [app] [ERROR]
    17:25:42.193 INFO  [app] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    17:25:42.193 INFO  [app] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    17:25:42.193 INFO  [app] [ERROR]
    17:25:42.194 INFO  [app] [ERROR] For more information about the errors and possible solutions, please read the following articles:
    17:25:42.194 INFO  [app] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
    17:25:43.675 INFO  [app] Service stopped
    17:25:44.748 INFO  [app] Service stopped
    [INFO] Tests run: 60, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.751 s - in io.quarkus.ts.many.extensions.DevModeManyExtensionsIT
    [INFO]
    [INFO] Results:
    [INFO]
    [INFO] Tests run: 120, Failures: 0, Errors: 0, Skipped: 0

Build of the TS takes 01:37 min without tests, should be faster for just 28 modules

mvn clean package -DskipTests takes 01:37 min on my MBP, it's third run to avoid spending time downloading.

There are few things to look into:

  • try to identify why it takes so long
  • try to optimize current situation
  • implement monitoring on size / build time characteristics / tests execution time / health of the testsuite
    • we have Jaeger tracing service, we should have daily jobs to push data to it and some (Grafana?) reports around that
    • new TS is crutial for us and we need to keep monitoring it
  • regularly evaluate coverage in the TS, structure of the modules, packages / TestCase / Test names

CC @Sgitario / @pjgg / @mjurc

mvn clean package -DskipTests
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Quarkus QE TS: Parent 1.0.0-SNAPSHOT:
[INFO]
[INFO] Quarkus QE TS: Parent .............................. SUCCESS [  2.498 s]
[INFO] Quarkus QE TS: Config .............................. SUCCESS [  6.126 s]
[INFO] Quarkus QE TS: http-minimum ........................ SUCCESS [  1.814 s]
[INFO] Quarkus QE TS: http-advanced ....................... SUCCESS [  6.900 s]
[INFO] Quarkus QE TS: Scaling tests ....................... SUCCESS [  2.147 s]
[INFO] Quarkus QE TS: Micrometer: Prometheus .............. SUCCESS [  4.323 s]
[INFO] Quarkus QE TS: Micrometer: Prometheus with Kafka ... SUCCESS [  7.369 s]
[INFO] Quarkus QE TS: Micrometer: OIDC .................... SUCCESS [  4.538 s]
[INFO] Quarkus QE TS: Messaging: Artemis .................. SUCCESS [  2.528 s]
[INFO] Quarkus QE TS: Messaging: Artemis + JTA ............ SUCCESS [  2.067 s]
[INFO] Quarkus QE TS: Messaging: AMQP with Reactive Messaging SUCCESS [  2.898 s]
[INFO] Quarkus QE TS: Messaging: QPID ..................... SUCCESS [  2.052 s]
[INFO] Quarkus QE TS: Messaging: Reactive Kafka ........... SUCCESS [  2.936 s]
[INFO] Quarkus QE TS: Messaging: Reactive Kafka with Avro . SUCCESS [  3.819 s]
[INFO] Quarkus QE TS: MicroProfile ........................ SUCCESS [  2.097 s]
[INFO] Quarkus QE TS: Security: Basic ..................... SUCCESS [  2.154 s]
[INFO] Quarkus QE TS: Security: HTTPS ..................... SUCCESS [  2.352 s]
[INFO] Quarkus QE TS: Security: JWT ....................... SUCCESS [  2.678 s]
[INFO] Quarkus QE TS: Security: Keycloak .................. SUCCESS [  2.410 s]
[INFO] Quarkus QE TS: Security: Keycloak + Authorization .. SUCCESS [  3.380 s]
[INFO] Quarkus QE TS: Security: Keycloak + JWT ............ SUCCESS [  3.240 s]
[INFO] Quarkus QE TS: Security: Keycloak + Webapp ......... SUCCESS [  5.027 s]
[INFO] Quarkus QE TS: Security: Keycloak + OAuth2 ......... SUCCESS [  5.035 s]
[INFO] Quarkus QE TS: Security: Keycloak + MultiTenant .... SUCCESS [  4.967 s]
[INFO] Quarkus QE TS: Security: Keycloak + OIDC Client .... SUCCESS [  3.722 s]
[INFO] Quarkus QE TS: SQL Database: Application ........... SUCCESS [  3.548 s]
[INFO] Quarkus QE TS: SQL Database: Multiple Persistence Units SUCCESS [  3.275 s]
[INFO] Quarkus QE TS: Quarkus CLI ......................... SUCCESS [  0.861 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:37 min
[INFO] Finished at: 2021-07-21T15:20:57+02:00
[INFO] ------------------------------------------------------------------------

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.