exasol / script-languages-release

Release Repository for Script Language Container for user defined functions (UDF's) that can be used in the EXASOL database.

EXASOL Script Languages

Please note that this is an open source project which is officially supported by EXASOL. For any question, you can contact our support team.

Overview

This project contains script-language containers for the user-defined functions (UDF's) that can be used in the EXASOL database (version 7.0.0 or later). Using script-language containers, user can extend the UDF's with additional libraries and tools, such as Python or R packages and their dependencies.

A script language container consists of a Linux container with a complete Linux distribution and all required libraries and a script client. The script client is responsible for the communication with the database and for executing the UDF code.

We provide in this repository several flavors of script-language containers, such as the current standard implementation of the script client with support for Python 2/3, R und Java. A flavor contains the build description of a container. We will show below how to customize and build the different script-language containers.

You can find pre-built containers in the release section of this repository. If you are interested in the script client you find more details here.

Features

• Build a script language container as docker images
• Export a script language container as an archive which can be used for extending Exasol UDFs
• Upload a script language container as an archive to the Exasol DB's BucketFS
• Generating the activation command for a script language container
• Can use Docker registries, such as Docker Hub, as a cache to avoid rebuilding image without changes
• Can push Docker images to Docker registries
• Run tests for you container against an Exasol DB (docker-db or external db)

Table of Contents

Information for Users

• User Guide
• Changelog

script-languages-release's Issues

Adapt build_steps to new exaslct analyze task

With exasol/script-languages@a9e90fe we improved the analyze task in exaslct. We need to adapt the build_steps of the standard and fancyr flavor to this analyze task.

SyntaxWarning: "is not" with a literal in docker_utils.py

Fix and check consequences, because it is actually a bug and logical error:

script-languages-release/exaslct_src/exaslct/lib/utils/docker_utils.py:0: SyntaxWarning: "is not" with a literal. Did you mean "!="?

https://github.com/exasol/script-languages/blob/15a491919956d2eb914c7f49932f43fd7299141e/exaslct_src/exaslct/lib/utils/docker_utils.py#L7

Java EMITS UDF Scripts cannot emit null

CREATE OR REPLACE JAVA SCALAR SCRIPT TEST.java_test() EMITS(i INT) AS 
class JAVA_TEST {
 static void run(ExaMetadata exa, ExaIterator ctx) throws Exception {
  ctx.emit(null);
 }
}
/

CREATE OR REPLACE PYTHON SCALAR SCRIPT TEST.python_test() EMITS(i INT) AS
def run(ctx):
 ctx.emit(None)
/

select test.java_test();   -- NullPointerException
select test.python_test(); -- works

Backtrace:

07.01 10:03:21.839 writeException: 22002 VM error:
java.lang.NullPointerException
Stack trace:
com.exasol.ExaIteratorImpl.emit(ExaIteratorImpl.java:73)
com.exasol.JAVA_TEST.run(JAVA_TEST.java:4)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:566)
com.exasol.ExaWrapper.run(ExaWrapper.java:186) (Session: 1655054869230996830)

Exclude unnecessary directories from /var during export

Situation:

Pull request #86 introduced that /var of flavor_customization gets copied to the release image. This is important for analyzing the images later. However, many things in /var are not necessary for the exported container and increase the size of the image by several megabytes.

Task:

Exclude the following directories during export

• /var/cache/apt
• /var/lib/apt
• /var/lib/dpkg

Documentation, section prerequisites

There should be a remark in the prerequisites section, that you need to execute the statements as root

Add regular CI tests for exaslct on Centos

Situation:

• Currently, we test exaslct only on Ubuntu, but some users want to use it under Centos 7 or 8

Tasks:

• Improve and extend https://github.com/exasol/script-languages/blob/1e0cac875a71d8da757ea14218a524454b1af3b6/test_exaslct_init_script.sh to allow multiple Guest OSes for the surrounding Docker Container
  • Maybe integrate it in the usual python unittests
• Add docker file for Centos with python 3.6
• Add different test scenarios
  • exaslct --help
  • exaslct build
  • exaslct export

echo_timestamp test might fail for very small difference in timestamps with R

The echo_timestamp test might fail for very small difference in timestamps with a R UDF. Reason unclear, but I suspect a rounding error.

Link to test: https://github.com/exasol/script-languages/blob/3806a3d418178a74143c562facfb82b73a3e9509/tests/test/generic/types.py#L153

Travis log: tkilias_script_languages_Job_179.46.txt

Add logo to Readme

ForkingPickler PicklingError under MacOSX with Python 3.8

Investigate if this can happen on Linux with Python 3.8:

ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'flavors_python3-my-flav.AnalyzeUDFClientDeps'>: import of module 'flavors_python3-.....

MacOSX is not supported and probably will never be supported.

EXAUDFCLIENT: Add alwayslink to cc_library and pin Bazel to version 1.0.0

See for more details here

UDFClient Instrumentation Wrapper

• Call instrumentation functions in the UDFClient Binary Wrapper bash script
  • Functions:
    • setup: e.g. generate output_filename
    • teardown: e.g. upload to results
    • wrap_execute: e.g. valgrind command
• Load the functions from a script in the bucketfs, no recompiling of the container
• Add to exeslct/test-framework functions which wrap the test
  • Functions:
    • setup: e.g. prepare socket
    • teardown: e.g. download results
• Add helpers for uploading stuff to the bucketfs which can be used in teardown
• Have a library of wrapper functions in bucketfs which can be activated by uploading a config file to the bucketfs
• Add wrappers for:
  • valgrind, callgrind, massif
  • stdout, stderr capture to bucketfs
  • profiler for java, python and c++ (gprof)
  • debugger
  • perf
  • strace

Add nightly CI jobs which rebuild the container without using the cache

Situation:

• Every so often, we encounter that the CI job for a commit on master fails because some dependencies are no longer available or are got incompatible
• Often, we do not recognize this problem before we merge
• Examples:
  • exasol/script-languages@1dcc358

Solution:

• Use Nightly builds to catch these situations for master

Alternatives:

• Pining Version to avoid incompatible version updates (does not solve the no longer available packages)
• Split components (UDFClient, exaslct and flavors) into different repos to test them separately and only use tested versions (does not solve the no longer available packages)

EXASLCT: Add build hook system

Situation:

• To replace the temporary solution of #99, we need build hooks for each build step in a flavor. The build hook need to run before all building the docker images and check if all packages in the package lists are available
• With this the build hooks are a different phase in the execution of the build then executing the image builds

pre-build-phase:   build-step-1_pre-hook -> build-step-2_pre-hook
build-phase:         build-step-1_build_image -> build-step-2_build_image
post-build-phase: build-step-1_post-hook -> build-step-2_post-hook, ...

The alternative would be hooks directly around the build_image task, but it triggers the image build together with the hooks, which do not want for the package check. We could allow the hooks to initiate a skip of the build step, but this feels hacky.

build-step-1_pre-hook -> build-step-1_build_image -> build-step-1_post-hook -> build-step-2_pre-hook -> build-step-2_build_image -> build-step-2_post-hook, ...

Maybe, both variants find a use case at some point, but for the moment different phases are the way to go.

Split standard flavor into one flavor for 6.2 and one for 7.0 and pin package versions

Situation:

• We have different package versions in the standard flavor for 6.2 and 7.0
• Package versions are currently not pined in the standard flavors
  Tasks:
• Released containers for their packages versions
• Split standard flavors
• Pin package version per standard flavor

Update package versions for standard flavors

Build returned following error messages:
pip-packages:

Starting with v0.37.0, the 'azure-storage' meta-package is deprecated and cannot be installed anymore.
Please install the service specific packages prefixed by azure needed for your application.
The complete list of available packages can be found at:
https://aka.ms/azsdk/python/all
Here's a non-exhaustive list of common packages:
- azure-storage-blob : Blob storage client
- azure-storage-file-share : Storage file share client
- azure-storage-file-datalake : ADLS Gen2 client
- azure-storage-queue: Queue storage client
(edited)
10:59

apt-packages:

E: Version '0.96.24.32.12' for 'software-properties-common' was not found
E: Version '1.1.1-1ubuntu2.1~18.04.5' for 'libssl-dev' was not found

Copy apt/dpkg package database into release image

Situation: currently we don't copy /var/lib/dpkg to the release image and as such we can't get a list of all installed ubuntu packages

Fix:
Copy /var/lib/dpkg to the final release image, but do not copy the cache

Add test for %env directive

Background

• We should test if the %env directive works for our UDFs, we already have some tests, but they don't cover all important environment variables See more here: https://docs.exasol.com/database_concepts/udf_scripts/programming_languages_detail.htm
• For example, LD_LIBRARY_PATH is not tests, to be precise the test is disabled
  • As more research showed, this test is disabled, because the docker-db has currently a bug
• However, we should also check other environment variables, such as
  • TZ (timezone)
  • LC_* (locale)
  • PATH

Acceptance Criteria

• Add tests for TZ, LC_* and PATH (the correct settings are unclear, so first return them and then lets discuss them)

Generate and store checksum of the exported container besides the container

Situation:

Sometimes uploads or downloads of the container can corrupt large files. To be able to check this, we need a checksum on original exported container. Furthermore, they also help to detect manipulations of containers.

Task:

Generate a checksum during export of a container and store it besides the container

Choice of checksum rationals:

• Because, we not only want to detect corrupted files, but also manipulated files we need a secure hash function where intentionally generating collisions is hard.
• We need a checksum where the corresponding tool is easily available and in the best case installed by default

Improve localhost resolution for running inside a docker container

We are already resolving localhost for external DBs to docker internal addresses, but this only for Linux. Add for the existing resolution support for MacOSX and add resolution for the upload command.

Resolution:
https://github.com/exasol/integration-test-docker-environment/blob/73b649af25ac19dffb6ef7119cf57c51d16ebe19/src/lib/test_environment/setup_external_database_host.py#L27

Localhost resolution for MacOSX:
https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds

Old version of bazel can not be installed from the apt repository

Version '1.0.0' for 'bazel' was not found from apt repository http://storage.googleapis.com/bazel-apt. The version actually exists in the repository, but is probably gets hidden by a newer one.

Solution: download dep-file directly from the Github releases

Investigate to run exalsct in docker container to better support MacOSX and Windows

Situation:

• exalsct needs a linux system to run properly
• a Linux VM is quite heavy to install for the build
• Maybe it is possible to start exalsct in a docker container with a linux system and mount the docker socket into this container. Then exaslct can spawn new container to build the script-language container

Todo:

• test docker-socket mounting on MacOSX and Windows
• prepare a container to run exaslct in
• change start up script of exaslct to run exaslct in the container and mount the socket

Add pull-request blocker which checks if the HEAD of submodules is at a ancestor of origin/master

Unable to load native library in Standard Flavor with Virtual Schemas and Exasol 6.2

For unknown reasons the Virtual Schemas don't work with the current master of the script-languages. The standard container delivered with Exasol 6.2 seems not to be affected. The error happens while the initialization of JVM, so only Java seems to be affected. The tests for the Java-UDFs were all green during build and worked.
My current guess is that something goes wrong during loading of JNI Libraries in some JDBC Drivers. An interesting fact is, that the JVM crashed during initialization. At this point, no user specific code should run. So, it might be some static loads in the jar files or something like that. It also might be correlated to the update of the Virtual Schemas to Java 9.

Here is the Output of the FlavorTest Build of the Standard Flavor:

27.1.0.10:33408> Changing to script directory /exaudf
27.1.0.10:33408> DBG: exaudfclient.cc(98) in main Load libexaudflib
27.1.0.10:33408> DBG: exaudfclient.cc(99) in main VAR: libexaudflibPath = [/exaudf/libexaudflib_complete.so]
27.1.0.10:33408> zmqcontainerclient argv[0] = ./exaudfclient
27.1.0.10:33408> zmqcontainerclient argv[1] = ipc:///tmp/zmqvmcontainer_conn_7830610158718577077
27.1.0.10:33408> zmqcontainerclient argv[2] = lang=java
27.1.0.10:33408> ### SWIGVM starting ./exaudfclient with name 'ipc:///tmp/zmqvmcontainer_conn_7830610158718577077 (1,5): 'ipc:///tmp/zmqvmcontainer_conn_7830610158718577077'
27.1.0.10:33408> DBG: exaudflib/exaudflib.cc(1554) in exaudfclient_main Reinit
27.1.0.10:33408> DBG: exaudflib/exaudflib.cc(96) in external_process_check VAR: &(socket_name_file) = [0x7f29a3aa60c0]
27.1.0.10:33408> DBG: exaudflib/exaudflib.cc(151) BEGIN FUNC: socket_send
27.1.0.10:33408> DBG: exaudflib/exaudflib.cc(192) BEGIN FUNC: socket_recv
27.1.0.10:33408> ### SWIGVM connected with id 6cabea7374fc69b5
27.1.0.10:33408> DBG: exaudflib/exaudflib.cc(151) BEGIN FUNC: socket_send
27.1.0.10:33408> DBG: exaudflib/exaudflib.cc(192) BEGIN FUNC: socket_recv
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(111) CALL BEGIN: setClasspath()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(111) CALL END: setClasspath()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(112) CALL BEGIN: getScriptClassName()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(112) CALL END: getScriptClassName()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(113) CALL BEGIN: importScripts()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(113) CALL END: importScripts()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(114) CALL BEGIN: addPackageToScript()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(114) CALL END: addPackageToScript()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(115) CALL BEGIN: addExternalJarPaths()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(115) CALL END: addExternalJarPaths()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(116) CALL BEGIN: getExternalJvmOptions()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(116) CALL END: getExternalJvmOptions()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(117) CALL BEGIN: setJvmOptions()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(117) CALL END: setJvmOptions()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(118) CALL BEGIN: createJvm()
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-Dexasol.scriptclass=com.exasol.adapter.RequestDispatcher]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-Xms128m]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-Xmx1024m]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-Xss512k]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-XX:ErrorFile=/tmp/hs_err_pid%p.log]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-Djava.class.path=/tmp:/exaudf/javacontainer/libexaudf.jar:/buckets/bfsdefault/postgres/postgresql-42.2.6.jar:/buckets/bfsdefault/postgres/virtualschema-jdbc-adapter-dist-2.0.1.jar]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(245) in createJvm VAR: options[i].optionString = [-XX:+UseSerialGC]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(255) in createJvm VAR: m_env = [0]
27.1.0.10:33408> DBG: javacontainer/javacontainer.cc(257) CALL BEGIN: int rc = JNI_CreateJavaVM(&m_jvm, (void**)&m_env, &vm_args)
27.1.0.10:33408> Error occurred during initialization of VM
27.1.0.10:33408> Unable to load native library: �?��)

Add cuda profiling library to cuda preview flavor

Currently, the cuda preview flavor misses the CUPTI library. Add it to the list of packages to install from the cuda sdk.

Add tools to test container

While debugging problems interactively, I missed several tools in the test container:

• vim (create sql files)
• ip (check ip adresses)
• ss (checking open ports)
• tmux

Fix that python-ds flavors get build instead pulled in the Google Cloud CI Release Job

Situation:

• We removed the python-ds flavors from the Google Cloud CI Jobs to save costs and time. The thought was that we already build them on the base repository, but that has different commit id, such that the Release Job doesn't find the images
  Options:
• Add the python3-ds flavors back to the Release Build Jobs of the Google Cloud CI
• Remove them from the list of flavors of the Release Job of the Google Cloud CI and release them on the base repository
• Fix exaslct that it can pull the target docker images as well, if source and target options are different, because the Release Job pulls the images with commit hash in the tag and renames them to release images

Pin package versions for flavors

• Standard flavor 7.0
• Standard flavor 6.2
• Python3-ds 6.1 cuda

Fix package install scripts for empty package lists

https://github.com/exasol/script-languages/blob/ce0e779385abedbba0ce54f5f3e866bd561744c8/ext/scripts/install_via_apt.pl
https://github.com/exasol/script-languages/blob/ce0e779385abedbba0ce54f5f3e866bd561744c8/ext/scripts/install_via_pip.pl
https://github.com/exasol/script-languages/blob/ce0e779385abedbba0ce54f5f3e866bd561744c8/ext/scripts/install_via_r_versions.pl

Error when emitting only one Timestamp column: AttributeError: 'int' object has no attribute 'isoformat'

Tried this using the Python3 dialect:
Emitting timestamp columns works fine, as long as these columns are not the only column that is emitted. If you try to emit only one timestamp column, an error message is thrown.
Example for emitting one column:

--/
CREATE OR REPLACE PYTHON3 SET SCRIPT test_udf(seconds int) EMITS (abc TIMESTAMP) AS

def run(ctx):
    import pandas as pd
    import numpy as np
    from datetime import datetime
    import io
    
    c1=np.empty(shape=(100),dtype=np.object_)
    
    c1[:]=str(datetime.now())
    
    df=pd.DataFrame({0:c1})
    df[0]=pd.to_datetime(df[0])
    output = io.StringIO()
    df.info(buf=output)
    ctx.emit(df)
/

select test_udf(100);

Resulting error message:

[Code: 0, SQL State: 22002]  VM error: AttributeError: Traceback (most recent call last):
  File "TEST_UDF", line 15, in run
  File "<EXASCRIPT>", line 132, in emit
AttributeError: 'int' object has no attribute 'isoformat'
 (Session: 1666104991743518184)

When other columns are added, it works:

--/
CREATE OR REPLACE PYTHON3 SET SCRIPT test_udf(seconds int) EMITS (abc TIMESTAMP, cde INT) AS

def run(ctx):
    import pandas as pd
    import numpy as np
    from datetime import datetime
    import io
    
    c1=np.empty(shape=(100),dtype=np.object_)
    c2=np.empty(shape=(100),dtype=int)
    c1[:]=str(datetime.now())
    c2[:]=1
    df=pd.DataFrame({0:c1,1:c2})
    df[0]=pd.to_datetime(df[0])
    output = io.StringIO()
    df.info(buf=output)
    ctx.emit(df)
/

select test_udf(100);

exaslct generate-language-activation generates a slash to much for standard-EXASOL-7.0.0

Situation:

exaslct generate-language-activation generate a slash to much between the path-in-bucket and the container-name.

Reason:

The template contains a slash to much.

https://github.com/exasol/script-languages-release/blob/master/flavors/standard-EXASOL-7.0.0/flavor_base/language_definition

Compare with template for standard-EXASOL-6.1.0
https://github.com/exasol/script-languages-release/blob/master/flavors/standard-EXASOL-6.1.0/flavor_base/language_definition

Change:

Fix template and think about tests

Reference EXA_toolbox.upload_github_release_file_to_bucketfs in documentation

Reference the script as an alternative, if you only need a prebuilt container and your cluster can access the network and has nameserver configured

https://github.com/exasol/exa-toolbox/blob/master/utilities/README.md#upload_github_release_file_to_bucketfs

Check alter session generation for all flavors

Situation:
Following path were returned by the alter session generation

zmq+protobuf:///bucketfs1/testbucket///standard-EXASOL-7.0.
zmq+protobuf:///bucketfs1/testbucket//standard-EXASOL-6.1.0

Todo:
Check the templates and the generation code for export and generate-language-activation

Improve Readme

• incorporate the changes from exasol/script-languages#57
• fix the git clone statement
• add ALTER SESSION example
• example for testing with external db and different docker-db version

R install_script using the versions package doesn't work anymore

Situation:

In nightly build rebuild/nightly_2020_09_24_01_22_28 we got for the flavors standard-600 and standard-700 the following error message.

Step #3 - "Build images": ERROR: DockerBuildImageTask_1f21cdc3c4(image_name=exasol/script-language-container:standard-EXASOL-6.0.0-flavor_customization): Exception in run: Error occurred during the build of the image exasol/script-language-container:standard-EXASOL-6.0.0-flavor_customization_YTHVT32C65FX55L2SF6AQTKXRHMUFHPIDUVMQYMUIVPALHZOAI6A. Received error "The command '/bin/sh -c /scripts/install_scripts/install_via_r_versions.pl --file /build_info/packages/flavor_customization/r_cran_packages --rscript-binary Rscript --with-versions --allow-no-version' returned a non-zero code: 1" .The whole log can be found in /workspace/.build_output/jobs/2020_09_24_10_38_56_DockerBuild/outputs/DockerBuild_79d97e690d/DockerFlavorBuild_587fb166ad/DockerCreateImageTaskWithDeps_be8e3046af/DockerBuildImageTask_1f21cdc3c4/logs/docker-build.log
Step #3 - "Build images":
Step #3 - "Build images": Failed to run Rscript -e 'library(versions);install.versions(c("stringr"),c("1.4.0"))' at /scripts/install_scripts/package_mgmt_utils.pm line 122.
Step #3 - "Build images":
Step #3 - "Build images": Execution halted
Step #3 - "Build images": incomplete final line found on '/tmp/RtmpFTiGZ9/file217aa3e2b0'
Step #3 - "Build images": 7: In readLines(file, encoding = "UTF-8") :
Step #3 - "Build images": no non-missing arguments to max; returning -Inf
Step #3 - "Build images": 6: In max.default(numeric(0), na.rm = FALSE) :
Step #3 - "Build images": incomplete final line found on '/tmp/RtmpFTiGZ9/file217cd9ee63'
Step #3 - "Build images": 5: In readLines(file, encoding = "UTF-8") :
Step #3 - "Build images": be detected
Step #3 - "Build images": The current version and publication date of stringr could not
Step #3 - "Build images":
Step #3 - "Build images": 4: In current.version(pkgs) :
Step #3 - "Build images": incomplete final line found on '/tmp/RtmpFTiGZ9/file217ea20a22'
Step #3 - "Build images": 3: In readLines(file, encoding = "UTF-8") :
Step #3 - "Build images": no non-missing arguments to max; returning -Inf
Step #3 - "Build images": 2: In max.default(numeric(0), na.rm = FALSE) :
Step #3 - "Build images": incomplete final line found on '/tmp/RtmpFTiGZ9/file2161129fe'
Step #3 - "Build images": 1: In readLines(file, encoding = "UTF-8") :
Step #3 - "Build images": In addition: Warning messages:
Step #3 - "Build images": Calls: install.versions -> version2date -> available.versions
Step #3 - "Build images": missing value where TRUE/FALSE needed
Step #3 - "Build images": Error in if (!all(df$available)) { :
Step #3 - "Build images":
Step #3 - "Build images": Executing: Rscript -e 'library(versions);install.versions(c("stringr"),c("1.4.0"))'
Step #3 - "Build images":
Step #3 - "Build images": ‘/tmp/RtmpUxzsTp/downloaded_packages’
Step #3 - "Build images": The downloaded source packages are in
Step #3 - "Build images":
Step #3 - "Build images":
Step #3 - "Build images": * DONE (versions)
Step #3 - "Build images":
Step #3 - "Build images": ** testing if installed package can be loaded
Step #3 - "Build images":
Step #3 - "Build images": ** building package indices
Step #3 - "Build images":
Step #3 - "Build images": *** installing help indices
Step #3 - "Build images":
Step #3 - "Build images": ** help
Step #3 - "Build images":
Step #3 - "Build images": ** preparing package for lazy loading
Step #3 - "Build images":
Step #3 - "Build images": ** R
Step #3 - "Build images":
Step #3 - "Build images": ** package ‘versions’ successfully unpacked and MD5 sums checked
Step #3 - "Build images":
Step #3 - "Build images": * installing source package ‘versions’ ...
Step #3 - "Build images":
Step #3 - "Build images":
Step #3 - "Build images": downloaded 8033 bytes
Step #3 - "Build images": ==================================================
Step #3 - "Build images": Content type 'application/x-gzip' length 8033 bytes
Step #3 - "Build images":
Step #3 - "Build images": trying URL 'http://cran.uk.r-project.org/src/contrib/versions_0.3.tar.gz'
Step #3 - "Build images":
Step #3 - "Build images": (as ‘lib’ is unspecified)
Step #3 - "Build images": Installing package into ‘/usr/local/lib/R/site-library’
Step #3 - "Build images":
Step #3 - "Build images": Executing: Rscript -e 'install.packages("versions",repos="http://cran.uk.r-project.org")'
Step #3 - "Build images":

Cleanup flavors

Situation:
The exported flavors get larger and larger with every package.

Task:

• Check if recommend packages get installed, if yes deactivate
• Check if other large files exists which can safely removed and remove them

Add docker image id to image_info after the image was created

• Add the image ID after the build to the image info such that the parent images can access it, either for checks or persisting it into their image for later debugging

Add a build breaker to the CI which fails if the commit was not rebuild without cache

Situation:

• Every so often, we encounter that the CI job for a commit on master fails because some dependencies are not available anymore or are got incompatible
• Often, we do not recognize this problem before we merge
• Currently, we need to rebuild the container without cache which takes very long and as such we can't use it for every commit
• Examples:
  • exasol/script-languages@1dcc358

Solution:

• Add CI triggers for rebuild jobs which react to the branch prefix rebuild/
• Add a build breaker to the CI which checks if a rebuild was triggered for the commit
  • We can use here the branch prefix rebuild/ as well

Caveats:

• The current build breaker will fail and succeed on the same commit and GitHub actions don't overwrite their previous result
• With this the last commit for every pull request will signal a failure. With this problem, this will only be a temporary solution.
• The final solution will be completely pinned package versions with a fast way to check if these versions still available and up to date.

Add boto3 to standard flavor

Collect the release images on docker hub for the Exasol releases

Situation:

• in our documentation we said that you should download the standard container from the bucketfs, import it to docker and compile your package within this container
• In future we want to provide the container of each Exasol Version as docker image at docker hub

Acceptance Criteria

• we need to upload the image during a release to docker hub with the right Exasol Version attached
• we need an own docker hub repository, such the user can easily browse and search for the Exasol version
  Open Questions:
• How do we do the release?
  • a normal release releases all flavors, so if the tag would be an exasol version it is confusing
  • could we use special tags for exasol standard container releases? Which only releases a specific flavor for a specific version.

Depends

Depends on:
exasol/script-languages-container-tool#86
exasol/integration-test-docker-environment#131

Future

#396

Add test which imports jars which load JNI libraries

Situation

• Some java libraries load native libs via JNI
• We have delicate of build of the Java Container and this might cause issues with loading native libs
• Many libraries distribute the native libs within the jar and extract it at runtime and it to the search path (see exasol/hadoop-etl-udfs#52)
• The later process is dangerous and often is based on JVM implementation details

Acceptance Criteria

• We need a test case for loading JNI files which are already extracted, and we can add them to java.library.path via the UDF pragma %jvmoptions
• We need a test case for adding an already native lib at runtime to the search path which is already present in the file system
• We need a test case with extracting the native lib from the jar and adding it at runtime to the search path
• For adding a native lib to the search path might exist multiple solutions, we need to search for the most commons ones and implement them

Add script to install specific version of a R packages

Add examples for installing packages as comments to flavor_customization dockerfiles

• installing R packages
• installing R packages with a specific version (requires #23)
• installing pip packages
• installing apt packages

EXASLCT: add pull, load or local only

Currently, builds Exaslct the images if it can't find, load or pull them. In the CI (especially in the Release Job of the Google Cloud CI) it is sometimes beneficial to use an already build and tested images. Then the better option is to fail instead of building the images. Add flags with which we can specify which options to get the image are allowed.

Build TLS certificates for RDS into Java truststore

refer to exasol/script-languages#56

Add pysftp and paramiko to standard flavor

Simplify the adaption of flavors for users

Situation:

• currently it is difficult for users to adapt the flavors, because they don't know how to write Dockerfiles, and we only provide a minimal explanation in the Readme

Todo:

• Add default install script to flavor_customization Dockerfile to install via apt, pip and R, such that the user only need to change the package lists
• Add a few examples how to write a Dockerfile into the flavor_customization Dockerfile
• Add a comment that the "From {image}" in the flavor_customization Dockerfile is needed and should be only change if you know what you are doing
• Add a comment to other build steps that you should only change them if you know what you are doing
• Add link to Dockerfile documentation

Add simplejson package to flavor standard-EXASOL-7.0.0

Add actual package lists to images at end of image building

Situation:

Currently, we only add the package lists for the install scripts to the images. These only contain the expected packages, but not the actual installed packages. Usually, the package lists for install scripts don't contain all packages, only those which must be in the container. However, package systems will install further package to resolve dependencies. With the list of actuall installed packages, we can better trace back what is in the container and can use diagnostic UDF to show what is available in an installed container. Furthermore, we can check if all packages got installed which is important for R because it can silently omit packages and it can be difficult to recognize such an error in the build logs.

Tasks:

• If not available create scripts which extract the actual installed packages and format them in the same format as the package lists for the install script, such that comparison is easy
• At the release step of the flavor run the scripts to extract the actual installed packages
• Compare the actual installed packages with requested ones and fail the build if requested once are not available. This requires that all flavors and steps use the new install script and their package list format. => Will be fixed with #557

Add exasol websocket-api for Python3 to standard flavor

Python3-ds-EXASOL-6.0.0 flavor_customization uses wrong python-binary in install script

Situation:

The install script for python3 package in python3-ds-EXASOL-6.0.0 flavor_customization uses python3 as binary. However, this is not available for on the Ubuntu 16.04 distribution and the alternatives python3 version installed from another source only provides a binary python3.6

https://github.com/exasol/script-languages/blob/8eaf9b54f571b564c67be3f968146541ecbcdcec/flavors/python3-ds-EXASOL-6.0.0/flavor_customization/Dockerfile#L23

Changes:

Correct parameter for install script to

RUN /scripts/install_scripts/install_via_pip.pl --file /build_info/packages/flavor_customization/python3_pip_packages --python-binary python3.6 --with-versions --allow-no-version