exasol / exasol-test-setup-abstraction-server

REST-API for exasol-test-setup-abstraction-java that allow using it from different languages

Exasol Test-Setup Abstraction GO

This project makes the features of exasol-test-setup-abstraction-java available in the following languages:

• Go

Usage

Go

package test

import testSetupAbstraction "github.com/exasol/exasol-test-setup-abstraction-server/go-client"

func myTest() {
	exasol, err := testSetupAbstraction.New().
                    CloudSetupConfigFilePath("myConfig.json").
                    DockerDbVersion("8.23.1").
                    StartupTimeout(time.Minute * 10).
                    Start()
	if err != nil {
		panic("failed to start test setup")
	}
	connection := exasol.CreateConnection()
	//...
}

Additional Information

• Changelog
• Dependencies
• Developers Guide

exasol-test-setup-abstraction-server's Issues

Startup fails when docker is not running on localhost

This is caused by exasol/exasol-test-setup-abstraction-java#33 and fixed in https://github.com/exasol/exasol-test-setup-abstraction-java/releases/tag/0.3.1. To fix this we need to upgrade to the latest version.

java.lang.IllegalStateException: E-ETAJ-12: Failed to ssh to the exasol database. This is required for redirecting a host port. Known mitigations:
* Make sure the database is reachable (port 22 open)
* Make sure 'ec2-user' can login to the database using your ssh-key.
        at com.exasol.exasoltestsetup.SshConnection.createSession(SshConnection.java:49)
        at com.exasol.exasoltestsetup.SshConnection.<init>(SshConnection.java:30)
        at com.exasol.exasoltestsetup.testcontainers.ExasolTestcontainerTestSetup.<init>(ExasolTestcontainerTestSetup.java:41)
        at com.exasol.exasoltestsetup.ExasolTestSetupFactory.getTestSetup(ExasolTestSetupFactory.java:48)
        at com.exasol.testsetupabstraction.server.Main.main(Main.java:16)
Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection refused (Connection refused)
        at com.jcraft.jsch.Util.createSocket(Util.java:394)
        at com.jcraft.jsch.Session.connect(Session.java:215)
        at com.exasol.exasoltestsetup.SshConnection.createSession(SshConnection.java:38)
        ... 4 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at java.base/java.net.Socket.connect(Socket.java:558)
        at java.base/java.net.Socket.<init>(Socket.java:454)
        at java.base/java.net.Socket.<init>(Socket.java:231)
        at com.jcraft.jsch.Util$1.run(Util.java:362)
        at java.base/java.lang.Thread.run(Thread.java:829)

panic: failed to start test-setup-abstraction server. Cause: exit status 100

Go client: Allow configuring the startup timeout

Sometimes startup fails with the following error message, e.g. for this build:

failed to start server. The server did not print a port number. Output: "", error stream: "2023-10-18 09:03:07.649 [INFO   ] Starting exasol test setup using dummy config file\n2023-10-18 09:03:07.807 [INFO   ] Using Exasol test container setup (auto mode with configuration file non-existing-config-file-1697619787800.json missing)\n2023-10-18 09:03:08.207 [INFO   ] Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')\n2023-10-18 09:03:09.204 [INFO   ] Found Docker environment with local Unix socket (unix:///var/run/docker.sock)\n2023-10-18 09:03:09.257 [INFO   ] Docker host IP address is localhost\n2023-10-18 09:03:09.288 [INFO   ] Connected to docker: \n  Server Version: 24.0.6\n  API Version: 1.43\n  Operating System: Ubuntu 22.04.3 LTS\n  Total Memory: 6921 MB\n2023-10-18 09:03:09.309 [INFO   ] Checking the system...\n2023-10-18 09:03:09.311 [INFO   ] ✔︎ Docker server version should be at least 1.6.0\n2023-10-18 09:03:09.315 [FINE   ] Exposing ports: [8563, 2580, 443, 20002, 22]\n2023-10-18 09:03:09.714 [INFO   ] Pulling docker image: exasol/docker-db:8.23.0. Please be patient; this may take some time but only needs to be done once.\n2023-10-18 09:03:10.525 [INFO   ] Starting to pull image\n2023-10-18 09:03:10.566 [INFO   ] Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)\n2023-10-18 09:03:10.773 [INFO   ] Pulling image layers:  5 pending,  1 downloaded,  0 extracted, (405 bytes/? MB)\n2023-10-18 09:03:11.367 [INFO   ] Pulling image layers:  4 pending,  2 downloaded,  0 extracted, (144 MB/? MB)\n2023-10-18 09:03:11.878 [INFO   ] Pulling image layers:  3 pending,  3 downloaded,  0 extracted, (297 MB/? MB)\n2023-10-18 09:03:11.897 [INFO   ] Pulling image layers:  2 pending,  4 downloaded,  0 extracted, (297 MB/? MB)\n2023-10-18 09:03:11.976 [INFO   ] Pulling image layers:  1 pending,  5 downloaded,  0 extracted, (310 MB/? MB)\n2023-10-18 09:03:15.713 [INFO   ] Pulling image layers:  1 pending,  5 downloaded,  1 extracted, (558 MB/? MB)\n2023-10-18 09:03:25.452 [INFO   ] Pulling image layers:  1 pending,  5 downloaded,  2 extracted, (1 GB/? MB)\n2023-10-18 09:03:28.007 [INFO   ] Pulling image layers:  1 pending,  5 downloaded,  3 extracted, (1 GB/? MB)\n2023-10-18 09:03:34.176 [INFO   ] Pulling image layers:  1 pending,  5 downloaded,  4 extracted, (1 GB/? MB)\n2023-10-18 09:03:34.210 [INFO   ] Pulling image layers:  1 pending,  5 downloaded,  5 extracted, (1 GB/? MB)\n2023-10-18 09:04:01.196 [INFO   ] Pulling image layers:  0 pending,  6 downloaded,  5 extracted, (3 GB/3 GB)\n2023-10-18 09:06:15.457 [INFO   ] Pulling image layers:  0 pending,  6 downloaded,  6 extracted, (3 GB/3 GB)\n2023-10-18 09:06:15.475 [INFO   ] Pull complete. 6 layers, pulled in 184s (downloaded 3 GB at 20 MB/s)\n2023-10-18 09:06:15.523 [INFO   ] Creating container for image: exasol/docker-db:8.23.0\n2023-10-18 09:06:15.744 [INFO   ] Container exasol/docker-db:8.23.0 is starting: ebb9241ec417c7e409592831633eaa4ecfcace56af3bf4aaf6e23d097aa1a29e\n2023-10-18 09:06:16.431 [FINE   ] No status cache found for container \"ebb9241ec417c7e409592831633eaa4ecfcace56af3bf4aaf6e23d097aa1a29e\". Creating fresh status.\n2023-10-18 09:06:16.432 [FINEST ] Waiting for cluster configuration to become available.\n2023-10-18 09:06:17.305 [FINEST ] Docker container supports docker exec\n"

The timeout of 500s = 8.3min seems to be too short in some situations, especially on slow GitHub runners. That's why we should make the timeout configurable for the Go client.

Allow creating connection without autocommit

Allow creating SQL connections with no autocommit

Allow getting host IP

When using exasol-test-setup-abstraction-server in integration tests where we need to access the Exasol database from another docker container, the connection info object returns exasolContainer.getHost() as host which usually is localhost. This name however won't work when trying to access Exasol from another docker container. In this case the exasolContainer.getHostIp() might be more helpful.

AC

• The SqlConnectionInfo object returned by getConnectionInfo() contains additionally the result of exasolContainer.getHostIp() in case of ExasolTestcontainerTestSetup.
• For StandaloneExasolTestSetup we need to find another value that makes sense in this case.

Workaround

Use hard-coded IP address 172.17.0.1.

Don't start a new container for each working directory

When used in a Go project like https://github.com/exasol/extension-manager/, the server starts a new Exasol container for each tested Go package which slows down tests significantly. The root cause is fixed in exasol/exasol-testcontainers#220, so we need to upgrade to the new release https://github.com/exasol/exasol-testcontainers/releases/tag/6.5.1

Make GetConnectionInfo() in Go client public

The GetConnectionInfo() method is useful when configuring the application under test with DB credentials.

Fix dependabot warning

Dependabot warns about a vulnerability in golang.org/x/net, see #33

Make Exasol V8 the default docker db database being used.

Start fails due to too short timeout

Response status not checked when downloading the Server jar

When the Go client downloads the Server jar from GitHub, it does not verify the response status. When the file does not exist, the Go client ignores the response code 404, writes an HTML error page to the jar and tries to execute it which fails.

Instead the Go client must fail with an error message if the status code is not 2xx.

Initial Implementation

add an initial implementation

Fix failing dependency check

 Error:  Failed to execute goal org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0:audit (default-cli) on project exasol-test-setup-abstraction-server: Detected 1 vulnerable components:
Error:    io.netty:netty-handler:jar:4.1.86.Final:runtime; https://ossindex.sonatype.org/component/pkg:maven/io.netty/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
Error:      * [CVE-2023-34462] CWE-400: Uncontrolled Resource Consumption ('Resource Exhaustion') (7.5); https://ossindex.sonatype.org/vulnerability/CVE-2023-34462?component-type=maven&component-name=io.netty%2Fnetty-handler&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1

Add project-keeper source for go

Add project-keeper source for go

Blocked by exasol/project-keeper#307

Fix license url for go-client

The go-licenses returns the following license URL for the go-client:

https://github.com/exasol/exasol-test-setup-abstraction-server/blob/go-client/v0.2.3/go-client/LICENSE

This URL is broken, because the LICENSE file is in the root folder. To fix this, we just need to copy the LICENSE file into the go-client directory.

Add API for accessing BucketFS

Add API for accessing BucketFS

Release tags don't work when using the Go client

When a projects wants to use the go client, it needs to specify a non-release version in go.mod, e.g. github.com/exasol/exasol-test-setup-abstraction-server/go-client v0.0.0-20220530070706-8388ba012403. Creating a tag v0.2.0 did not help.

Acceptance Criteria:

• It's possible to reference the go client with a release version like v0.2.0
• Path https://pkg.go.dev/github.com/exasol/exasol-test-setup-abstraction-server/go-client?tab=versions lists the release version
• The project readme contains a badge for the go package: https://pkg.go.dev/badge/?path=https%3A%2F%2Fpkg.go.dev%2Fgithub.com%2Fexasol%2Fexasol-test-setup-abstraction-server%2Fgo-client

Notes

A solution could be this: https://go.dev/doc/modules/managing-source#multiple-module-source

Remove workaround for target dir bug

Once exasol/exasol-test-setup-abstraction-java#46 is fixed we can remove the workaround.

Fix CVE-2024-29025 in `io.netty:netty-codec-http:jar:4.1.107.Final:runtime`

Error:    io.netty:netty-codec-http:jar:4.1.107.Final:runtime; https://ossindex.sonatype.org/component/pkg:maven/io.netty/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
Error:      * [CVE-2024-29025] CWE-770: Allocation of Resources Without Limits or Throttling (5.3); https://ossindex.sonatype.org/vulnerability/CVE-2024-29025?component-type=maven&component-name=io.netty%2Fnetty-codec-http&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1

Parallel use not possible

go by default runs the tests in parallel.
That makes trouble with this tool.

Problems:

• Name conflicts (tests using the same objects name clash, this could be solved by using unique schema names)
• Missing isolation, e.g. EXA_ALL_SCRIPTS contains objects from other tests
• Issues with the exasol-test-setup-abstraction-server: the download of the server jar is triggered by the first test. The second test tries to use the unfinished jar.

Idea for a solution:

Add an OS level mutex (a file or something similar). Using a GO mutex does not help since go test runs the different suits in dedicated processes.

Workaround

Start tests with option -p 1.

Allow overriding default Docker image

The server uses the default docker image configured in exasol-testcontainers. To allow using a custom docker image, the go-client must start the server with system property com.exasol.dockerdb.image if a custom image is specified.

Fix vulnerabilities in org.apache.commons:commons-compress:jar:1.24.0:compile

 Error:  Failed to execute goal org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0:audit (default-cli) on project exasol-test-setup-abstraction-server: Detected 1 vulnerable components:
Error:    org.apache.commons:commons-compress:jar:1.24.0:compile; https://ossindex.sonatype.org/component/pkg:maven/org.apache.commons/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
Error:      * [CVE-2024-25710] CWE-835: Loop with Unreachable Exit Condition ('Infinite Loop') (8.1); https://ossindex.sonatype.org/vulnerability/CVE-2024-25710?component-type=maven&component-name=org.apache.commons%2Fcommons-compress&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
Error:      * [CVE-2024-26308] CWE-770: Allocation of Resources Without Limits or Throttling (7.5); https://ossindex.sonatype.org/vulnerability/CVE-2024-26308?component-type=maven&component-name=org.apache.commons%2Fcommons-compress&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1