exasol / extension-manager

Exasol extension-manager

Exasol Extension Manager

Extension Manager Java Client: , Extension integration tests library:

This project contains the Exasol extension manager. It's a tool for installing and managing extensions like Virtual Schemas.

REST API Documentation

The extension-manager exposes a REST API for the frontend.

This API is documented via Swagger. In order to view it, checkout this repo, run go run ./... and open http://localhost:8080/swagger/index.html.

Additional Information

• Changelog
• Software Design (online, main branch)
• Software Design (local)
• Dependencies

Developer Guides

• Guide for embedding Extension Manager in another application
• Guide for developing an extension for Extension Manager
• Guide for developing Extension Manager

Related Projects

• extension-manager: Extension manager backend written in Go (this repo)
• extension-manager-interface: Extension interface defined in TypeScript, published to npm as @exasol/extension-manager-interface
• extension-parameter-validator: Validator for extension parameters written in TypeScript, published to npm as @exasol/extension-parameter-validator
• Virtual Schemas providing extensions
  • s3-document-files-virtual-schema: Work in progress, see PR#84

extension-manager's Issues

Add endpoint for getting parameters

When users list the installed extensions they don't need to know the parameters. In order to keep the response small, we should

• remove the parameters them from the list
• and provide a separate endpoint GET /api/v1/extensionmanager/installations/{id}/{version} that returns the parameters.

This should be implemented before #65

Add OpenApi documentation for parameters

Parameters follow the structure described in the interface.
To simplify work for client implementors we need to document this also in the OpenAPI spec.

This should be implemented after #67 is done.

Support multiple API versions

Currently the extension manager supports only extensions with a single API version (SupportedApiVersion), i.e. a version of https://github.com/exasol/extension-manager-interface/

After it a new API is released, updating all extensions will be a lot of effort. That's why the extension manager should support a list / range of versions.

See #28 (comment)

Acceptance criteria

• Integration tests verify, that all supported API versions actually work

Requirements and design are outdated

• Ensure that requirements and design are up-to-date
• Add missing coverage tags
• Enable requirements tracing in CI

Add interface

Add an interface for extensions

Add error handling

Add error handling to the extension api
For

• SqlClient
• for methods like install

Currently requests return an 500 Internal server error also for missing parameters (e.g. dbHost URL param). A helpful error message and a code 4xx must be returned in this case.

Add REST endpoint for getting extension parameters

Add an REST endpoint for getting extension parameters

Extension discovery

Make the extension-manager automatically pick up extension definition files.

Refactor URLs of the REST API requests

To have a uniform API we need to use the following request URLs:

GET /api/v1/extensionmanager/extensions (List)
PUT /api/v1/extensionmanager/extensions/{id}/{version}/install
DELETE /api/v1/extensionmanager/extensions/{id}/{version}

GET /api/v1/extensionmanager/installations
POST /api/v1/extensionmanager/installations/{id}/{version}/instances
GET /api/v1/extensionmanager/installations/{id}/{version}/instances
DELETE /api/v1/extensionmanager/installations/{id}/{version}/instances/{id}

Fix golangci-lint issues

When running golangci-lint with option --enable-all it finds many warnings in the source code. We should fix these warnings (or configure exclusions) and use --enable-all on the CI build.

We enabled the following linter presets: bugs,complexity,format,import,module,sql,comment,performance,test.

Presets error,style,metalinter,unused are still disabled because they cause too many false positives.

Add design

add a design

Check preconditions of extensions

Check the preconditions (required JAR files) of the extensions during GetAllExtensions.

Allow embedding into other backend

EM needs to be embeddeble into other backends and contribute it's requests via the go-rest OpenApi.

Add requirements and design document

Add requirement tracing with OFT.

Add support for extension context

Add support for extension context as implemented in exasol/extension-manager-interface#19

Document meaning of status 404 for GetExtensionDetails request

When GET /api/v1/extensionmanager/extensions/{extensionId}/{extensionVersion} returns status 404 it means that it's not possible to create an instance of the extension because it's not supported.

We need to document this in the OpenApi specification of the request.

Add design for test-framework for extensions

We need to provide a test-framework for the extensions.

Ensure thread safety for go interpreter

See https://github.com/dop251/goja#is-it-goroutine-safe

Is it goroutine-safe?
No. An instance of goja.Runtime can only be used by a single goroutine at a time. You can create as many instances of Runtime > as you like but it's not possible to pass object values between runtimes.

Add prefix to log messages from extensions

JS Extensions can log via console.log(). These log messages should be prefixed with the extension id. This will simplify debugging because it's easy to see which extension wrote a log message.

Handle panics in validator JavaScript code

When validating parameters, EM executes JS code. When this code throws an Error (e.g. because of a bug), this will cause a panic in Go. In order to avoid crashing the server, we need to handle panics from the JS code.

Create Plugin Development Documentation

At the moment there is no documentation how to implement a Custom Plugin.
It would be good if we have a documentation for it and a separate Github Repository with a sample plugin.

Maybe a good feature for us would be that a plugin needs is own repository and the name needs to start with exasol-extension
Like exasol-extension-aws-vs-s3.
This is used by Terraform for discovering community plugins, maybe we want something similar.

Refactor transaction handling

Currently extensions are responsible for committing the transaction.
This should be done by EM itself.
EM must also check that the extension does not execute transaction management commands like commit or rollback and cause an error.

Uninstall an extension

Allow uninstalling an extension in a certain version

Add tool for generating REST API documentation

A proper API documentation will help the other teams to use this client.
Hopefully there is something similar to swagger for go.

Fix failing dependency check build

Dependency check fails with the following error message:

 Error:  Failed to execute goal org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0:audit (default-cli) on project extension-manager-client-java: Detected 1 vulnerable components:
Error:    com.fasterxml.jackson.core:jackson-core:jar:2.14.0:compile; https://ossindex.sonatype.org/component/pkg:maven/com.fasterxml.jackson.core/[email protected]?utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
Error:      * 1 vulnerability found (7.5); https://ossindex.sonatype.org/vulnerability/sonatype-2022-6438

Specify port via command line switch

Currently the extension manager runs on port 8080. To make this more flexible we should add a command line switch (e.g. --port 8080) that allows specifying a custom port.

The integration test framework then should be updated to use specify this argument instead of reading the port from the extension manager's output.

See exasol/s3-document-files-virtual-schema#84 (comment)

Use error-reporting-go

Use error-reporting-go for unified error codes.

Precondition: Error code crawler for Go.

Verify files in BucketFS via hash sum

Currently extensions verify their required files in BucketFS only via the file size in bytes. We should improve this by verifying the hash sum (e.g. sha256) instead.

This will require changes to

• extension-manager-interface
• https://github.com/exasol/extension-manager
• each extension

See exasol/s3-document-files-virtual-schema#84 (comment)

Add parameter validator

Move API to dedicated repo

The extension API has a different release cycle. For that reason we should move it to a dedicated repo. However, we did not do it's quite unstable until now and that would require a release for each change,

Add request for installing extensions

Extensions expect that the EXA_EXTENSION schema exists when the extension manager installs them.
So EM must create the schema before running the installation if it does not yet exist.

Add getExtensionParameters endpoint

Add more Exasol metadata tables

Implement interface specified in exasol/extension-manager-interface#17

Add backend

Add a first implementation of the backend

Move Go sources to pkg directory

To better group Go sources we should move them to a common pkg directory.
This will require updates to package imports.

Move go code from backend folder to project root

since it's the only code in the repo now, the nested folder is no longer required

Handle errors in JavaScript code

The JavaScript code of extension definitions potentially throws errors in case of programming errors. This causes the code to panic which crashes EM.
That's why we need to gracefully handle errors in JavaScript code.

Implement listing and deleting instances

Allow listing and deleting instances of an extension, e.g. virtual schemas.

Create infrastructure for extension registry

The extension registry is a json file that lists all available extensions.
We need to deploy an HTTPS endpoint that allows the extension manager to download the registry file.

Use matching extension manager version in integration tests

Currently integration tests always use the latest extension manager version. This is a problem for these reasons:

• This will break existing tests for extensions when a new, incompatible version of the extension manager is published.
• You want to test an extension against a specific previous release of extension manager.

Instead of using version latest the tests should use the EM version they where built against. We can implement this by using the current project version as default instead of latest in method ExtensionTestConfig.getExtensionManagerVersion().

• Verify consistent versions in .project-keeper.yml and pom.xml (properties/revision)
• Use current version for installing EM

Verify version in swagger annotation

verify the version in the swagger annotation in restApi.go

Add createInstance endpoint

Add an REST API endpoint for creating extension instances, validate parameters

Allow configuring the path to the extension directory

Currently the path to the extension directory is hard coded. We need to allow specifying a custom directory at startup.

Add REST API

Add a REST API

Mark extension versions as "latest" or "deprecated"

This will help users decide which version of an extension to install.

Siehe ExtensionsResponseExtension / InstallableVersions boolean flags deprecated & latest.

Allow extensions to forward error messages from extensions to the user

Some errors that are thrown in Extension must be reported to the user. Currently all errors thrown in JavaScript code are reported as 500 Internal server error. Extensions must be enabled to customize status code and error message.

Example: parameter validation failed.

Add integration test framework for extensions

Developers who create extensions need to add integration tests. To simplify writing these tests we need to create and publish a framework for extension integration tests.

We can use the source code at s3-document-files-virtual-schema as a starting point.

Return status 404 for unknown extensions

Currently EM fails with status 500 for an unknown extension ID. Instead it should return status 404.

Allow testing extensions with node

When loading an extension that calls registerExtension(createExtension()) in a Jest test, tests fail when loading the module because installedExtension is not defined:

    ReferenceError: installedExtension is not defined

      43 | }
      44 |
    > 45 | registerExtension(createExtension())
         |                  ^
      46 |

      at registerExtension (node_modules/@exasol/extension-manager-interface/src/api.ts:242:23)
      at Object.<anonymous> (src/extension.ts:45:18)
      at Object.<anonymous> (src/extension.test.ts:2:1)

A possible solution is to set global.installedExtension in registerExtension().

See also exasol/extension-manager-interface#12

Optionally load extensions from URL

Loading extensions via HTTP would simplify deployment.