Background

• one of the parts of the Framework will be a python event handler. Usually, the logic of the event handler is implemented by the user, but we need to implement the Framework to execute this event handler.
• The Framework is responsible for the loading and storing the state, for call the user provided event handler and managing temporary tables
• To test event handler a python implementation of the event loop is useful, because debugging with the Lua event loop is probably difficult
• Users need to be able to specify the class which implements the event handler logic
  • Alternative, they might define a UDF from which this class is imported. This is for adhoc algorithms, when we don't want to build a python package or language container for the algorithm.

Acceptance Criteria

• Create Python Event-Handler Framework
• Create Python Event Loop for Testing

Background

• The ResultType of the UDFQueryHandler assumes a Dict[str,Any] as return value and it expects that this will be serialized to json from the QueryHandlerRunnerUDF
• However, currently, we transform the result into a string and return this instead

Background

• the functions in the query_loop module are big and complex, we should extract function with proper names #47 (comment)
• currently, we also use quite a lot of magic number, replace them with constants #47 (comment)

Background:

Some algorithms need communication between UDF Instances of a UDF Call for example ML algorithms or some HPC algorithms. To achieve this, we need first to discovery all UDF instances of a UDF call. We can have many instances in a cluster, such that we need to do first a local discovery per node.

• In case of any errors in the execution, temporarily created
  -- bucketfs files
  -- tables
  should be cleaned, if with the keeping option not selected.

• don't let Lua do sys.exit() when an error occurs

Background

• The framework should help with certain task and we need to document how the framework can help with them and what our assumptions are

Acceptance Criteria

• Add a design document for the Framework
• Use OpenFastTrace https://github.com/itsallcode/openfasttrace

Background

• #47 (comment)
• #47 (comment)
• #47 (comment)
• #47 (comment)
• #47 (comment)

• in the current installation, the bundled Lua script is generated while installing the project

• however this installation requires Lua and its dependencies to be installed in the user machine.

• in order to break this dependencies, we should store the bundled Lua scripts in the wheel.

• add functionality to save bundled script

• provide an option to cli whether to re-bundle or not

• call re-bundling in each commit to be sure that it is saved

• add a temporary UDF stub
• add deployment scripts
• prepare test fixtures

Background

• The handle_finish call at the following position might throw exceptions by itself and might hide the Exception that caused it.

  advanced-analytics-framework/exasol_advanced_analytics_framework/testing/mock_query_handler_runner.py

  Line 47 in dc9b6c7

  self.handle_finish()

Background

    def _invalidate(self):
        self._check_if_valid()
        self._invalid_object_proxies = self._invalid_object_proxies.union(self._valid_object_proxies)
        self._valid_object_proxies = set()
        self._owned_object_proxies = set()
        self._is_valid = False
        child_context_were_not_released = False
        for child_query_handler_context in self._child_query_handler_context_list:
            if child_query_handler_context._is_valid:
                child_context_were_not_released = True
                child_query_handler_context._invalidate()
        if child_context_were_not_released:
            raise RuntimeError("Child contexts are not released.")

Background

• It is difficult to identify which query_handler_context wasn't released, if we get an Exception. To allow better diagnostics, we should store the stack trace during the constructor call of the query_handler_context as a string inside the query_handler_context.

Background

• currently we need a UDF mock or a UDF to test the EventHandler
• For complex EventHandler this makes testing unnecessary hard
• we should implement a MockEventContext which we can create from a column definition and a List of tuples

Background

• We recognized in downstream projects, that mocking some parts was really difficult
• Specifically mocking the Exasol Lua Script-specific part, such as pquery
• For that reason, we decided to inject these parts at the top

Background

• Currently, we silently release an object of a child EventHandlerContext in a parent EventHandlerContext
• However, this indicates a missing release call of the child context at as this an error

Background

• with poetry 1.4.0 it doesn't create the setup.py anymore https://github.com/python-poetry/poetry/releases/tag/1.4.0
• we currently use poetry build to generate the setup.py
• however, the setup.py isn't needed with newer pip versions and if there are releases to pypi or as wheels
• for that reason, we can remove setup.py and githook that generates it, from this repo

Acceptance Criteria

• #94
• Update workflows to poetry 1.4.0
• Remove setup.py Github Workflow
• Remove setup.py githook
• Remove setup.py file

Background

• In real life, we need to expect to lose sometimes messages, for that reason, we need to design and test our protocols in a way that they can survive this
• ZMQ ROUTER and DEALER use tcp, but also there exists failure scenarios, especially because the send operation is asynchronous and we don't get informed about reconnects
• This abstraction helps us to test our protocols with lost messages

Acceptance

• Add ZMQ Wrapper which injects faults by losing messages on random

Background

• The names EventHandler and EventLoop are a bit misleading, because we don't actually handle arbitrary events. We basically only react to the results of a query
• QueryHandler and QueryLoop hopefully represent better the actual purpose

Background

• To test SQL-based query handlers we need to actually run the query handlers against a database, however using the Lua-based QueryHandlerRunner requires the build of the language container
• To avoid this, we need a Python-based runner which can send the queries to the database via a SQL-Client
• But furthermore, by abstracting the SQL-Client, we can also mock this and compare in unit test the generated queries with our expectation.

Background:

I saw we are missing a README in this project.

Acceptance Criteria

• Add a README in the usual style, but add that this repository is in early development

Background

• From work consists of an Event Loop and an Event-Handler
• The Event Loop calls the Event Handler and executes queries the Event Handler returned
• The Event Loop needs to be written in Lua, such that we can run in a Lua Script to use pquery to run the queries
• The Event loop needs to receive configuration and must forward it to the event handler

Acceptance Criteria

• Implement Event-Loop as Lua Module
• Implement Lua Script
• Implement deployment cli

• add gtihub workflows
• add githooks
• a short README.rst
• doc folder including changes folder, guides... etc
• poetry setup
• subdirectories such as lu/src, lua/test

• We need a custom EventHandler class inside language container to test EventLoop.
• However, it is not desired to put the test codes into release package.
• Therefore, we need to have an additional language container flavor for testing purpose.
• The repo architecture could be as follows:

- pyproject.toml
- exasol_advanced_analytics_framework
- tests/test_package/
	- pyproject.toml
		- include: ../event_handlers or if this doesn't work we create symlink ../event_handlers -> test_package
- tests/event_handlers
	- test_event_handler1.py
	- test_event_handler2.py
- language_container
	- exasol_advanced_analytics_framework
		- flavor_base
			- release
				- install exasol_advanced_analytics_framework
			- test depends on release
				- install test_package

Background

• exasol/data-science-utils-python#60 extracted the interfaces from the DBObjectName, such as TableName, ...
• For more complex garbage collection of DBObjects we need to handle the DBObjectProxies like normal DBObjectNames, such that we can combine them with the DBObject classes and to be able to track dependencies between them.

Acceptance Criteria

• Rename DBObjectProxies to DBObjectNameProxies
• Let DBObjectNameProxies implement the DBObjectName Interfaces

Background

• At the following code position, it looks like, we only remove the object from the current context, if the other context is not a child (means a sibling.). However, we need to remove the object regardless of the relationship from the owner list of the current context,
  

  advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/top_level_query_handler_context.py

  Line 142 in 0947079

  self._remove_object(object_proxy)

Background

• currently we still use Lua 5.1, because some parts of the dev environment use Ubuntu 20.04

Acceptance Criteria

• update dev environment (CI, vagrant)
• update code (use const)
• update exaerror

Background

Vagrant is a tool for developer to provision reproducible development environments. We use vagrant to provide a developer sandbox for testing this project.

Acceptance Criteria

• Add a vagrant setup similar to https://github.com/exasol/machine-learning-library/tree/main/vagrant

Background

• Currently, the implementation directly uses ZMQ.
• However, this complicates unit tests.
• To simplify unit tests for the protocol, we need to replace the zmq usage with an abstraction.

Background

• Most algorithms are formulated as imperative programs with data flow parts
  • This means the high-level structure of the algorithm consists of for-loops, while-loops, if-statements and sequences of statements.
  • The single statements are then often data flows
• Data flows are usually represented in this framework as SQL queries
• Because only Lua Scripts can run dynamic SQL queries in the same transaction and we wanted to have the core logic in Python, we designed this framework as event driven, where the event handler returns the queries which Lua will execute
• However, implementing complex algorithms with an event handler is fairly complex. State Machines get easily very complex, and Hierarchical State Machines are not so much more helpful.
• To simplify the implementation of algorithms, we need a framework which allows implementing an imperative program where the statements are event handler. This allows the main logic of the algorithms in a very familiar paradigm and only on the lowest level, we need to care about the event handling.

Design Idea

class Variables

    def __init__(parent_scope_variables)
        """
        Variables are scoped and typed and need to be declared before usage. The scope means when ever you entry a Block (e.g. in IfStatement, your variables get a new Scope, variables you create in this scope are only visiable in this scope or subscope, but not in the parent scopes.)
        """
        self.parent_scope_variables = parent_scope_variables
        self.local_variables:Dict[name,Variable] = {} 

    def declare(name, type, value)
    
    def get(name)
    
    def set(name, value)


class Statement
        """
        We implement objects for the usual imparative constructs, such as Statements, If-Statements, Loops, .... Except that our statements consist of an init_handler method which is called first and handle_event method, which is can execute SQL queries via the Continue Result
        """
    def init_hanler(variables:Variables)
    def handle_event(event_context, event_handler_context):Union[Continue,Finished]
    
class StatementChain(Statement)

    def __init__(event_hanlder_factories:List[StatementFactory])
        self.event_hanlder_factories=event_hanlder_factories
        
    def init_hanler(variables:Variables)
        self.event_handlers = create_event_hanlder(self.event_hanlder_factories)
        self.current_event_handlers_idx = 0
        self.event_handlers[self.current_event_handlers_idx].init_handler(variables)
        
    def handle_event(event_context, event_handler_context):Union[Continue,Finished]
        result = self.event_handlers[self.current_event_handlers_idx](event_context, event_handler_context)
        if isinstance(result, Continue):
            return result
        else
            if self.current_event_handlers_idx<len(self.event_handlers)-1:
                self.current_event_handlers_idx+=1
                self.event_handlers[self.current_event_handlers_idx].init_handler(variables)
            elif
                return result
                
class ForEachStatement(Statement)

class WhileStatement(Statement)

class IfStatement(Statement)

see https://github.com/exasol/udf-mock-python/blob/487edc2488781e764770b4fd4debc1d3132ea4d7/exasol_udf_mock_python/mock_meta_data.py#L37

Background

• BundleLuaScripts uses hard-coded parameter
• query_loop_main.lua contains code which should be part of query_loop.lua, because they are specific to the query loop and are needed in other project if they use the query loop
• We would like to reuse it in other projects

Acceptance Criteria

• Refactor BundleLuaScripts such that we can reuse it
• Move _prepare_init_query and dependent function to query_loop.lua

Background

• We need to generate Cleanup Queries for EventHandlerContext in reverse order of creation, because objects might depend on each other, such that we need to remove them in the right order to avoid failure due to dependent objects.
• For example, views might depend on other views or tables

Background

• With #79 we implemented the UDF Discovery on a single node.
• Now we need to implement the discovery between nodes.

Background

We need a User Guide for this repo. It should contain the following sections:

• Setup
• Usage
• Implementation of algorithms

Background

• This is, currently, the hash function of DBObjectNameProxy

    def __hash__(self):
        self._check_if_valid()
        return hash(id(self))

Background

• Currently, the EventHandlerContext only provides an interface to create and load temporary bucketfs files, but didn't have yet a way to clean up
• Implementing the EventHandlerContext as a scope-based system gives us finer control over when temporary objects get cleaned up. This is important, in case we execute a long sequence of operations, where each operation can create temporary objects. At some point, some temporary objects are not needed anymore and can be release. Without that, we would potentially accumulate many large tables.
• In the scope-based EventHandlerContext, a parent event handler creates a ScopeEventHandlerContext and forwards it to a child event handler. The child event handler uses the EventHandlerContext for creating temporary objects. If the child event handler finishes, the parent releases the ScopeEventHandlerContext and with that, all temporary objects created by the child event handler.
• We can, of course, build the scopes recursively, where all intermediate event handlers get ScopeEventHandlerContext from which they can create new ChildEventHandlerContext.
• On the top layer we have a special EventHandlerContext which finally can clean up the release objects after the event handler returned
• Furthermore, we need to be able to transfer temporary objects from one scope to another scope, in case an event handler returns it or gets it as input. For this, a temporary objects needs to be owned by exactly one ScopeEventHandlerContext and we need a way to transfer ownership between the ScopeEventHandlerContext (parent, child, sibling)
• Also, if a ScopeEventHandlerContext forgets to release its temporary objects, they should get released by its parent ScopeEventHandlerContext. This ensures, at least at the end of the event_handler we clean up.

Background

• We plan for the framework a Lua loop which executes SQL queries and runs an Event Handler which consume their result and issues new SQL Queries
• The Event handler should if possible be written in Python for that we need to start a Python UDF which can have a bit of overhead.
• Furthermore, the Event handler is stateful, that means in each call to the UDF we need to store the event handler in the BucketFS and load it during the next call.
• The total overhead should be best below 0.5 seconds or at least below a 1 second
• We also need to know how fast the distribution of the event handler state in the bucketfs is for cluster with more cores

Background

• In the QueryHandlerContext and the ObjectProxies, we use currently _is_valid to show whether they are already released or not. We should unify the wording here to "released" or "not_released".
• Location with wrong naming:

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/top_level_query_handler_context.py

    Line 75 in 81d2513

    self._valid_object_proxies: Set[ObjectProxy] = set()

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/top_level_query_handler_context.py

    Line 76 in 81d2513

    self._invalid_object_proxies: Set[ObjectProxy] = set()

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/top_level_query_handler_context.py

    Line 189 in 81d2513

    def _check_if_valid(self):

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/top_level_query_handler_context.py

    Line 80 in 81d2513

    self._is_valid = True

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/top_level_query_handler_context.py

    Line 193 in 81d2513

    def _invalidate(self):

    (naming here could be a bit difficult _release(...) probably won't do it, because this function is for handling child contexts during release)

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/proxy/object_proxy.py

    Line 6 in 81d2513

    self._is_valid = True

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/proxy/object_proxy.py

    Line 8 in 81d2513

    def _check_if_valid(self):

  • advanced-analytics-framework/exasol_advanced_analytics_framework/query_handler/context/proxy/object_proxy.py

    Line 12 in 81d2513

    def _invalidate(self):

Background:

    def cleanup(self):
        if self._is_valid:
            raise Exception("Cleanup of BucketFSLocationProxy only allowed after release.")
        for file in self._bucketfs_location.list_files_in_bucketfs(""):
            self._bucketfs_location.delete_file_in_bucketfs(file)

Background

_ScopeQueryHandlerContextBase.get_temporary_path should be private because it is used by other functions and doesn't returna object that can be cleaned up

Background

• Our UDF Communication algorithm need to work under message loss. ZMQ ROUTER and DEALER use tcp, but also there exists failure scenarios, especially because the send operation is asynchronous and we don't get informed about reconnects
• To test the fault tolerance of our algorithms, we need to inject send faults

Acceptance Criteria

• Add a wrapper for the SocketFactory which can injects send faults
• Use the wrapper in the tests for the peer handshake

Background

• We have huge generated files for the Lua script checked into the repo, because we want to be able to install it without having lua amlg available
• However, it makes reviewing unnecessary hard
• Github seems to provide a way to hide this file https://github.com/github/linguist/blob/master/docs/overrides.md#summary
• It is likely that the following line in the .gitattributes file should help (please check, what file extension the jinja sql templates have)

*.sql linguist-generated

Background

• The framework should help with certain task, we need to document them and why we choose these

Acceptance Criteria

• Add a system-requirements document to the project
• Use OpenFastTrace https://github.com/itsallcode/openfasttrace

Background

• currently, we have only the handle_event method, which accepts a EventContext and a EventHandlerContext.
• however, for the first iteration of the EventHandler there is exists no Input for the EventContext, because we didn't execute a ReturnQuery yet
• for the first iteration we only have the input parameters
• For that reason, it is probably better to use two different methods for these cases

def handle_start(parameters,EventHandlerContext)->EventHandlerResult

def handle_query_result(EventContext)->EventHandlerResult

Background

• Currently, query_handler can't access the connection object to retrieve credentials.
• UDFs provide the get_connection function for this, we need to forward this to the query_handlers
• Because, we already inject the query_handler_context and it is more or less for responsible for the communication with the outside, it is probably the best point to add this functionality.

Background

• In #53 we moved the serialization and deserialization into wrapper QueryHandler, but it seems we forgot to remove the json.loads in QueryHandlerRunnerUDF
• It also looks like, that the AAF_RUN_QUERY_HANDLER UDF which runs the QueryHandlerRunnerUDF is a scalar UDF and as such doesn't allow next, which is used by the query_result

Background

The EventContext is an abstraction around the UDFContext or any exasol driver (e.g. pyexasol). We simplified its interface, such that either of them can implement that. However, in contrast to the UDFContext returns next method the row and not a bool for indicating if there are more rows.

Acceptance Test

• fix bug
• add type hints to EventContext
• add tests

Background

• We need to remove the setup.py, before that, we need to release the project to pypi
• Here is an example workflow for this https://github.com/exasol/error-reporting-python/blob/79aca5e435de66792150da6e365eee3d0778a237/.github/workflows/release.yaml#L25
• @redcatbear or @kaklakariada have to add the organization secret to this project

Acceptance Criteria

• Extend the release workflow
• Add the organization secret