citrusframework / yaks Goto Github PK

We should reimplement Camel-K steps in the citrus manner.

For example Then integration gate-keeper should print ${citrusVariable} is not supported.

We should use a digest mechanism as in camel k to re-execute the tests that are changed.
Otherwise there's no way to re-run tests from the OpenShift console.

When a test uses additional components that are not part of the default YAKS runtime (e.g. an additional camel component like camel-groovy) we need to load the dependency and its transitive dependencies at YAKS runtime.

E.g. We can export the Camel K integration generated by syndesis, tell how to deploy camel-k-operator + yaks-operator, how to install the integration (+ related resources) and the tests and show the result.

I've already created the yaks org on docker hub. Ask me for credentials.

We should add some scripts to automate the release.

Add support for both OpenAPI v2 and v3

yaks report -o junit produces invalid xml file.
Yaks generated output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="org.citrusframework.yaks.JUnitReport" errors="0" failures="0" skipped="0" tests="2" time="0">
  <testcase name="integration.feature:3" classname="classpath:org/citrusframework/yaks/integration.feature:3" time="0"></testcase>
  <testcase name="integration.feature:7" classname="classpath:org/citrusframework/yaks/integration.feature:7" time="0"></testcase>
</testsuite>

But the valid output should be something like (missing <testsuites> tag):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
	<testsuite tests="43" failures="2" time="4797.518" name="github.com/apache/camel-k/e2e">
		<properties>
			<property name="go.version" value="go1.14"></property>
		</properties>
		<testcase classname="e2e" name="TestAddons" time="506.930"></testcase>
		<testcase classname="e2e" name="TestAddons/master_works" time="424.130">/testcase>
	</testsuite>
</testsuites>

See https://llg.cubic.org/docs/junit/

When user provide custom glue code in custom .jar files that are added to the YAKS runtime we need to auto detect the custom glue code packages. We need to add all package names to the YAKS runtime in order to be able to use the custom glue code in a test.

Add some lookup mechanism to find packages that hold custom step definitions as glue.

Support Groovy steps that enable the user to

• run Citrus test actions as script
• add Citrus endpoint configuration as script

Something like:

Given endpoint configuration
    """
    citrus {
      endpoints {
        http {
          server('helloServer') {
            port = 18080
            autoStart = true
          }
        }
    }
    """

Given create actions basic.groovy
    """
    run {
      echo('Hello from Groovy script')
      sleep().seconds(1)

      createVariables()
          .variable('foo', 'bar')

      echo('Variable foo=${foo}')
    }
    """
Then apply basic.groovy

Wait for Http URL steps should also support relative resource paths so the user does not have to always specify the full request URL

Before:

Then wait for URL http://localhost:${port}/todo to return 200 OK

After:

Given URL http://localhost:${port}
Then wait for path /todo to return 200 OK

Steps of different kind occasionally need to share Citrus components and configuration. For instance someone could write a custom step extension and needs to access the standard Http steps provided by YAKS in order to send/receive data via Http.

YAKS needs to make sure that steps can load components and configurations and share those with others. Citrus already provides a context that can be used in this manner. The Citrus context is able to bind/load components and other shared instances. This context is a per test suite storage. There is also a test context that is shared only within a single test case.

Running test with custom step does not work - just uploading gives the same result.

Steps to reproduce
following example

yaks test extension.feature -u steps/
2020/05/07 17:30:40 log.go:70: snap/installer "level"=0 "msg"="Snap is already installed: skipping"
2020/05/07 17:30:40 log.go:55: snap/java-deployer "level"=0 "msg"="Executing maven release phase on project steps/"
Error: error while creating deployment for source code: exit status 1

Unfortunately I don't know where to check for more logs (the operator pod doesn't show anything)

I would like to have support for the environment variables in feature file:

1. propagate YAKS_NAMESPACE var to the running pod (we can reuse https://github.com/citrusframework/yaks/blob/master/pkg/cmd/test.go#L101)
   Example:

Scenario:
  Given URL: http://user-report-system.${YAKS_NAMESPACE}.svc.cluster.local
  .....

2. support custom properties:

Scenario:
  Given URL: http://user-report-system.${CLUSTER_WILDCARD_DOMAIN}
  .....

Yaks command:

yaks test mytest.feature -e CLUSTER_WILDCARD_DOMAIN=apps.myocp.com

I would like to have an option for specifying custom maven repository.
yaks install --maven-repository=....

We should provide steps to create/consume cloud events via Knative eventing. The user should be able to connect to the Knative eventing broker via Http in order to create event data. In addition to that we need a way to consume events. In a first version we can focus on creating an Http service via Knative serving that gets triggered by Knative eventing.

The tests should be able to send/verify cloud event data structure in version 1.0. As usual event data payloads are validated with the usual message content validation mechanisms in Citrus.

Users can run bash scripts configured in yaks-config as part of a setup procedure before tests get executed.

The script paths are relative to the initial test directory that is run via yaks CLI. If the initial test directory has nested directories with scripts defined in yaks-config the relative paths are broken.

We run a normal JVM process (using TestRunner main class) when executing tests in YAKS S2i image. Need to change this to a Maven build in order to support dependency resolving mechanisms. This is mandatory when using components in tests that are not baked into the YAKS Docker image.

This helps us to support additional runtime dependencies that we need to add to the YAKS runtime. For instance when the test uses a camel component that is not part of the default YAKS runtime we want to automatically download the artifacts and transitive dependencies at runtime.

Dear Citrus/YAKS Contributors,

thank you very much for your efforts and ideas - highly appreciated. The approach of yaks combined with the possibilities of citrus seems really promising to me.
My potential use case is testing of camel-k routes. After reading through the documentation (citrus, citrus-simulator and yaks) I have three open points - or missing links (pls let me know if you prefer having each of them in a separate issue)
Situation: I have a camel-k route to test which looks like this:

from(fromEndpoint)
.to(toEndpoint)

In my case fromEndpoint is mostly an imap server or kafka and toEndpoint is database or kafka

I very much like the idea of using predefined gherkin steps in order to test my routes. And with the message steps combined with camel-k steps you provide a very powerfull pre-built solution. However these are the things where I would need some help/hints
Questions:

1. How do I configure the endpoint in a gherkin message step in YAKS? Via config? Via custom steps and adding my EndpointConfig.java there?
2. In my scenario, the routes normally listen to the fromEndpoints. So I need to mock those endpoints. As far as I understand the closest to this would be citrus-simulator. But this will not work for mail and kafka, right? What would be the best approach here: Looking for/building docker images that provide these endpoints like e.g. greenmail and deploy them during test with a citrus kubernetes client?
3. the toEndpoints recieve messages from the route, which seems to be well covered by citrus and is mentiond by Christoph here:
   All outbound calls to 3rd party services can be mocked with frameworks like Citrus.
   However in the camel-k case these endpoints need to be available as service in the cluster (if I understand correctly). How could I achieve this? Again with a citrus kubernetes client?

Thanks and regards - Boris

Citrus provides waitFor actions to check for an Http URL to return 200 OK for instance. Add this to the standard steps in YAKS so people can just use it

I am getting this error while rerunning the test.

1. I have the failed test:

----------------------------FAILED TEST -------------
Test suite results:
        test/UserReport.feature: Test Failed

2. Then I want to rerun - getting this error

➜  camel-k-example-event-streaming git:(add_tests) ✗ yaks test test/UserReport.feature
Test suite results:
        test/UserReport.feature: Operation cannot be fulfilled on tests.org.citrusframework.yaks "user-report": the object has been modified; please apply your changes to the latest version and try again

3. Try to rerun again - working

➜  camel-k-example-event-streaming git:(add_tests) ✗ yaks test test/UserReport.feature
test "user-report" updated
----------------------------CONTINUE AS EXPECTED -------------

Hi,

I am working on adding arm64 support in camel-k project, which is using yaks binary. It will be very helpful if the yaks arm64 binary is released.

I have successfully built and tested yaks on my local arm64 server.

Please let me know if any help is required. I am happy to help with this.

Part of #55, depends on #57

See also #59

Let's add to the yaks-config.yaml file the possibility to run scripts pre and post executing the command:

config:
  namespace: {}
pre:
- script: file.sh
- run: |
    echo Start
post:
- run: |
    echo End
    echo Bye
- run: echo Bye! 

Let's assume linux/osx for the moment (which is also the standard environment in pipelines).

Pre and post are two array of object that can be of type script or run:

• script references an executable file in a path that can be relative to the config file location
• run is basically a sh script that can be executed directly. Under the hood we may materialize it into a real sh file in the os tmp dir and run it using the config file directory as context

Pre are executed before all tests in the test group. Post after.

Camel-K integration is running step is bound only to the pod phase. The pod phase does not reflect container statuses. Let's see snipped from the documentation: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/

phase: running: The Pod has been bound to a node, and all of the Containers have been created. At least one Container is still running, or is in the process of starting or restarting.

This means that if our pod has more containers (knative sidecars or so) Camel-K probes are completely ignored by YAKS (at least one container is running - the sidecar -> the pod is ready)

We should monitor also pod conditions that bring finer granularity.

Part of #55

When doing yaks test file.feature, yaks now runs a single test (even with multiple scenarios) and prints the result. Exit code is 0 or 1 depending on the result.

When running yaks test dir, yaks should lookup all .feature files in the directory and execute the test. No matter if a single test fails, it should proceed with the others. Execution (for now) can be linear: one test after another.
When all tests are finished, it should print the result of each test. Exit code should be 1 if at least a test failed, 0 otherwise.

Create BDD test scenarios from Open API specification. All operations and paths mentioned in the swagger file should be covered in generated tests.

Use Yaks CLI and/or Maven plugin to generate the tests.

Part of #55, depends on #57

See also #59 and #60

We should add the possibility to run tests recursively in sub-dirs.

A test group can contain the following config:

config:
  recursive: true

When it's enabled (I'm not sure about the default value for it):

• .feature files present in the dir are still treated as tests belonging to the group and run
• sub-directories are treated as sub-test-groups to run (i.e. we should run them before or after the standard tests, using their own yaks-config.yaml config file).

Possible usages.

Basic

• dir1 (recursive)
  • test1.feature
  • dir2
    • test2.feature
    • test3.feature

Execution is: test1, test2, test3 in the current ns (not deleted after run).

SingleTestNS

• dir1 (recursive, temp-ns)
  • test1.feature
  • dir2
    • test2.feature
    • test3.feature

Execution is: test1, test2, test3 in a temporary namespace, deleted after run

MultipleTestNS

• dir1 (recursive, temp-ns)
  • test1.feature
  • dir2 (temp-ns)
    • test2.feature
    • test3.feature
  • dir3
    • test4.feature

Execution is:

• test1 in tempns1
• test2, test3 in tempns2
• test4 in tempns1 (same as test1)

Both temp ns deleted after run.

Yaks is not able to merge ENV properties provided by default with properties specified by -e option.

Current doc is for developers.

We should write a basic user doc that explain options available to standard users (BDD test writers + behavior coders).

YAKS is a tool for integration testing. Integration testing means that you could test integration between various systems and it can happen that you will need credentials to log in. We can introduce new section in yaks-config.yaml for loading properties from ConfigMap/Secret resource.

apiVersion: v1
data:
  secret-saas.properties: mySaaSCreds
kind: Secret
metadata:
  name: secret-saas
type: Opaque

config:
  namespace:
    temporary: true
  secret: secret-saas
  run: |
    oc create secret generic mock-secret-saas --from-file=secret-saas.properties -n ${YAKS_NAMESPACE}
   .....

Background: 
    Given URL: ${url_loaded_from_secret/ConfigMap}
    And HTTP request header Authorization is "Bearer ${token_from_secret}"
    And HTTP request header Content-Type is "application/json"
    And HTTP request body:  {  "Name" : "username" }

  Scenario:
    Given integration salesforce-to-workday is running
    When send POST /services/data/v20.0/sobjects/Account
    Then receive HTTP 201 Created
    And integration salesforce-to-workday should print Name: username

We should improve the JMS connection factory instantiation in a way that the user gives key-value pair properties in order to define the connection factory.

Given JMS connection factory
  | type      | org.apache.activemq.ActiveMQConnectionFactory
  | brokerUrl | tcp://localhost:61616

The idea is to have a look at the connection factory type information first and delegate the connection factory instantiation to a helper that is able to map properties to constructor arguments.

We can add more helpers for different connection factories over time and if we have an unknown connection factory we can try to auto guess the constructor by just using the order of properties given as a fallback

Provide steps to access an Open API specification and use the information to generate test data for tests.

Error occurred when adding to Yaks to Tekton.

[test-hello : test-hello] panic: user: Current requires cgo or $USER set in environment
[test-hello : test-hello] 
[test-hello : test-hello] goroutine 1 [running]:
[test-hello : test-hello] github.com/citrusframework/yaks/pkg/client.getDefaultKubeConfigFile(0x111f25b, 0x14da260)
[test-hello : test-hello]       /home/runner/work/yaks/yaks/pkg/client/client.go:149 +0xe0
[test-hello : test-hello] github.com/citrusframework/yaks/pkg/client.GetCurrentNamespace(0x0, 0x0, 0x111f2a0, 0xc000400200, 0xc0005dfd00, 0xc0005dfd00)
[test-hello : test-hello]       /home/runner/work/yaks/yaks/pkg/client/client.go:157 +0x601
[test-hello : test-hello] github.com/citrusframework/yaks/pkg/cmd.(*RootCmdOptions).preRun(0xc0000b8200, 0xc000314500, 0xc00019e9a0, 0x1, 0x1, 0x0, 0x0)
[test-hello : test-hello]       /home/runner/work/yaks/yaks/pkg/cmd/util.go:12 +0x55
[test-hello : test-hello] github.com/spf13/cobra.(*Command).execute(0xc000314500, 0xc00019e960, 0x1, 0x1, 0xc000314500, 0xc00019e960)
[test-hello : test-hello]       /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:805 +0x56b
[test-hello : test-hello] github.com/spf13/cobra.(*Command).ExecuteC(0xc000314280, 0xc0000b8080, 0xc000314280, 0x0)
[test-hello : test-hello]       /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
[test-hello : test-hello] github.com/spf13/cobra.(*Command).Execute(...)
[test-hello : test-hello]       /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
[test-hello : test-hello] main.main()
[test-hello : test-hello]       /home/runner/work/yaks/yaks/cmd/manager/main.go:42 +0x178

Add predefined BDD steps for

• Creating a Camel context
• Creating Camel routes (XML, Groovy, ...)
• Start/stop Camel routes

in the yaks-config.yaml file.

Glue packages holding predefined Cucumber step definitions in YAKS are hardcoded in YAKS runtime at the moment. We need a way to dynamically load the glue packages in order to not always load everything and in order to support custom glue code packages sitting in foreign .jar files added to the runtime.

The junit output is pretty simple:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="org.citrusframework.yaks.JUnitReport" errors="0" failures="0" skipped="0" tests="2" time="0">
  <testcase name="integration.feature:3" classname="classpath:org/citrusframework/yaks/integration.feature:3" time="0"></testcase>
  <testcase name="integration.feature:7" classname="classpath:org/citrusframework/yaks/integration.feature:7" time="0"></testcase>
</testsuite>

Would it be possible to change testcase name to the scenario name?

Citrus already provides basic Http steps to access an endpoint URL in a BDD Gherkin scenario. We need to enhance that functionality in order to send some POST requests with given payload and verify the outcome other than just the Http status code.

We agreed in issues #55 to have the following format for yaks-config files:

config:
  some-general-config: tbd
pre:
- script: ../prepare.sh
- run: |
    kamel version

    kamel install
    kamel run xxx.groovy
post:
- run: echo "Hello!"

https://github.com/citrusframework/yaks/blob/master/pkg/cmd/test.go#L580 wraps multiline commands to the yaks-script-*.sh file.

This approach breaks windows support because windows is not able to execute .sh file with #!/bin/bash (https://github.com/citrusframework/yaks/blob/master/pkg/cmd/test.go#L586) directive.

Maybe we should reconsider our decision with bash scripts and go on the Tekton pipeline way as @christophd suggested.

To avoid issues, it may be better to avoid executing all tests in parallel and linearize their execution.

Part of #55, depends on #57

Let's introduce a yaks-config.yaml file that, when present in a directory that is going to be run, can specify some options:

config:
  namespace:
    temporary: false
    autoRemove: true

The "temporary" option is set to false by default, "autoRemove" is true.
When the "temporary" option is set to true, before running the tests in the test group:

• A new namespace with a random name (starting with yaks-) is created
• Yaks is installed in that namespace with default options

After running all tests in the test group:

• If "autoRemove" is enabled (default), the temporary namespace is deleted
• Let's make sure that autoRemove only removes temporary namespaces if created

At the moment we can handle only one single Http client/server component when writing tests with YAKS. It should be possible to have multiple components and interact with these in a single test.

Something like:

Given Http client fooClient
Given Http client barClient

When active Http client is fooClient
And send GET /todo
Then receive Http 200 OK

When active Http client is barClient
And send GET /todo
Then receive Http 404 NOT FOUND

Looking back at the idea of externally configuring the environment and then run the tests with YAKS, I've noticed that I end up each time writing a lot of bash scripts to create resources and then use YAKS only to check that a pod prints a log line. Which is really not what I expected at the beginning.

I was wondering if we can take a different approach and allow creating resources e.g. in the background part of a test. With a bit of YAML you can really create complex things on Kubernetes, so this approach could help also make a test file also self-contained, which in turn will help also other IDE/Web tooling that we may be creating.

Something like:

Background:
  Given Kubernetes resource
  """
  kind: Xxx
  apiVersion: xxx/v1alpha1
  spec:
    data: yyy
  """

People would use this to create generic stuff for which we still don't have a specific DSL in YAKS. Wdyt?

For connecting tests with a database we need BDD steps to execute SQL statements and verify the result sets if any.

Citrus as a framework already provides the verification of SQL results sets where users can expect columns and values.

We need to add some BDD Gherkin steps that the user can add to the test in order to fetch data from a JDBC datasource and verify its outsome

Supposing we're going to do a hackathon on this (...), I imagine someone wants to write a java library that adds a specific behavior that is not present by default.

We can provide a nexus to publish artifacts.. but best if it works also with snapshots..

We can show how to run a test completely from the Openshift console if we create a manifest for the operator.

We can publish it internally to a quay account in order to show it on OCP4.

Users can add environment variables with CLI options

$ yaks test --env CITRUS_SPRING_JAVA_CONFIG=org.my.config.MyEndpointConfig

Users should be able to add those variables to the yaks-config.yaml configuration file

I've recently learned about Jitpack, which provides a handy way to use binary snapshots by directly referencing sources stored on Github.

I think that can provide a way to reference custom steps and glue code without having to start a Minio as we do now. We can still keep the minio approach for local development.

Wdyt?

We are still struggling with native pre/post script execution on the client station. Users are forced to have installed all tools/commands used by the script. Scripts have to be platform-independent that can be a big pain.

Discussed with @christophd , @nicolaferraro, and @VratislavHais we decided that we will move script execution to the YAKS container in order to unify the environment.

it will allow to help users editing yaks-config.yaml files.

registering it on https://www.schemastore.org/json/ would enable it in several IDEs by default.

I was discussing with @bouskaJ about strategies that can be used to build a test suite with YAKS. For simple cases, it's enough to delegate to a pipeline the creation of an environment, then execute the YAKS tests and report status. But for more complicated scenarios we should add some hooks to allow people to execute some pre/post steps before running the tests.

I was thinking to create a concept of test group. The group is materialized into a directory on the fs.

When running the group (directory), yaks executes tests for all feature files present in the directory.

But the group can also have a yaks-config.yaml file, with a structure that we can define.
An example of file can be (I'm borrowing something from Github actions):

yaks-config.yaml

config:
  some-general-config: tbd
pre:
- script: ../prepare.sh
- run: |
    kamel version

    kamel install
    kamel run xxx.groovy
post:
- run: echo "Hello!"

This is enough generic to plug whatever we want to use as install mechanism for resource, including helm or operator hub.

Aside from that, we can also have additional options in the config yaml. Like:

• an option to run it in a temporary namespace
• an option to execute all child groups (subdir) defined in a group

With these building blocks, we can allow creating a whole test suite. Even converting most of the Camel K e2e tests to use yaks.

Wdyt @christophd ?