However, when we used it there we recognized that the columns functions returns a Dict instead of a column object and that the mock implementation are only available in the test of this project, but are also needed in other projects
Acceptance Criteria
Let the columns method of the SQLExecutor return a column object
Move the MockSQLExecutor to a testing package which will be included in the package
Our current Schema classes have some issues with their implementation of __hash__ and are, in fact, only immutable structs, this is perfect for dataclasses.
The DBObjectName classes should be fine and should be normal classes.
We upload and download the base models from the BucketFS
The BucketFS synchronization is asynchronous, such that it can happen that we try to download a base model in CombineToVotingRegressorUDF before it was synchronized to the particular node where the UDF runs
Acceptance Criteria
Add retry and wait mechanism and parameter to CombineToVotingRegressorUDF
Currently, the these classes are implementations, however, in the AAF, we need new implementation of them which work as proxy to for the temporary dbobject managed by the QuerHandlerContext
Further, the AAF needs also classes for UDFs and ConnectionObjects
Acceptance Criteria
Extract Interfaces from the dbobject name classes
Add a static create method to the interfaces which create the default implementation
Remove the old Builder classes, because they are actually not so useful for python with its keyword arguments
Currently, we use the base-path specified by the model_connection to store the models
However, it is likely that we can't create bucketfs connections at will, for security reason, maybe someone creates one for us with the credentials and grants us the usage.
Acceptance Criteria
Add a bucketfs path to the model and use it to store the models
There is a dependabot alert for upgrading Pillow library to version 9.1.1
When reading a TGA file with RLE packets that cross scan lines, Pillow reads the information past the end of the first line without deducting that from the length of the remaining file data. This vulnerability was introduced in Pillow 9.1.0, and can cause a heap buffer overflow.
Opening an image with a zero or negative height has been found to bypass a decompression bomb check. This will now raise a SyntaxError instead, in turn raising a PIL.UnidentifiedImageError.
Currently, the ColumnPreprocessorCreator is more or less hard-coded
It combines the SQLPreprocessor with the creation of the ColumnTransformers in a hard-coded way
It uses a fixed strategy to decide which preprocessing is applied to which column
It only hard-codes the preprocessor MaxMinScaler and OneHotEncoding
It currently, assumes you use the source table as input for training
Acceptance Criteria
Build abstraction which combines SQLPreprocessors and ColumnTransformers creation
Make the ColumnPreprocessorCreator configurable using mapping between column selectors and this abstraction
Make the ColumnPreprocessorCreator return the table to use for further usage, to allow the training on globally modified data (which is necessary for Target Encoding)
-> This requires that the ColumnTransformers can have to modes, one for fit and one for transform
-> This requires that the PartialFitIIterator switch the mode to fit and the other iterators to transform
Currently, the PartialFitRegressorUDF is hard-coded in TrainRunner and TrainUDF
However, there might be different implementations for it. For example, you could have one with configurable epochs (already exists), early stopping, ...
Acceptance Criteria
Use a factory for the creation of the PartialFitRegressorUDF in TrainRunner and TrainUDF
For SGD based algorithms, you need to reshuffle the data each epoch
However, shuffling usually needs the whole dataset, which is not feasible in the UDF
A non-perfect shuffling (usually enough for SGD), can be done with limited memory
It works similar to Reservoir Sampling, except if the random oracle decides to that you replace a value in the memory with the current new value, you replace the old value, but also yield it to the iterator user
Acceptance Criteria
Implement a ReservoirShuffleIterator with limited memory which can be used for batches or rows in form of pandas dataframes