exasol / python-extension-common

Common functionality, shared by Exasol Python Extensions.

Exasol Python Extension Common

A package with common functionality, shared by Exasol Python Extensions, e.g.

• transformers-extension
• sagemaker-extension

Features

A deployer for script language containers (SLC) to be used by UDF-based extensions of Exasol database requiring a special SLC.

More documentation

• User Guide
• Developer Guide
• User Defined Functions (UDF)
• Script Language Containers (SLC)

python-extension-common's Issues

Make printing the activation commands optional

We don't always want to see the activation commands printed in the console. In particular, in AI-Lab they may confuse the user. She may think that they are still part of the installation routine, while the AI-Lab runs the session level activation when it opens a DB connection.

• There should be a parameter that turns the printing of the activation commands on/off. The default value should be ON.
• When printing the messages, only the session level activation command should be printed in full. Then there should be a warning, that applying a similar activation command at the system level may deactivate languages that were activated since this message was printed.

Start using a new interface in the bucketfs-python

Use the PathLike interface to connect to the bucketfs and upload SLC there.

Accommodate both on-prem and SaaS bucketfs.

Investigate how to delete an SLC uploaded by an integration test

Multiple integration tests use the same instance of the DockerDb. Most tests upload an SLC to the bucketfs of the ITDE. Currently, we have interference between tests, as the containers uploaded by previous tests are not removed.

Removing a container has side effects.

• When a file gets deleted, then uploading another file with the same name immediately afterward fails. This means there should be a delay between running subsequent tests if they use the same bucketfs file name.
• A language container is an archive, that gets unzipped in the bucketfs. Apparently, the deletion of the tar file causes the deletion of the unzipped directory. This needs to be confirmed and verified. We need to find out if this has an impact on the duration of the required delay between tests.

PEC: Add comment on using fixtures from pytest-plugin pytest-exasol-saas

Add comment to tests and developer guide about using fixtures from pytest-plugin pytest-exasol-saas

Acceptance criteria

• Comment in test_saas/conftext.py
• Section in doc/developer_guide/developer-guide.rst

Remove the code that cuts off an archive extension, once this code is adopted in the bucketfs-python

This is the code that should be deleted:

ARCHIVE_EXTENSIONS = [".tar.gz", ".tgz", ".zip", ".tar"]

    for extension in ARCHIVE_EXTENSIONS:
        if bucket_file.endswith(extension):
            bucket_file = bucket_file[: -len(extension)]
            break

PEC: Update to Python 3.10

Update

1. minimum Python version in file pyproject.toml
2. dependencies poetry update
3. Matrix build definitions in files in .github/workflows: Proposal: Update PTB and re-generate workflows
4. actions/checkout to v4 in GitHub workflows

Add click prompts for the cli security credentials parameters

The options' attributes should be set as it is described in the click options API documentation

Create the documentation

Initially, the language container deployment documentation can be taken from the Transformers extension repo.

PEC: Use plugin pytest-saas

• Replace dependencies to git repositories by dependencies to pypi packages
• Add dependency to pytest-exasol-saas
• Replace fixtures

Add integration tests for the SLC deployment

Migrate the SLC deployer integration tests from the TE or the SageMaker extension. Deploy a container from the SLC-release.

Add a function validating the completion of the SLC deployment.

The function should try to create a UDF and run it at all nodes. These steps need to be repeated until success or timeout.
Since the success is not 100% reliable proof that all files have been extracted from the archive, (some packages, not used by the UDF, may yet be extracted) the function should take a fixed time pause at the end.

The LanguageContainerDeployer should use this function in its container uploading process.