koi-learning / koi-api

The backend of the KOI-system.

Welcome to project site of the KOI-System

koi-api's Issues

copying instances

It should be possible to copy an already trained instance.
This would allow reusing a training state for debugging and development.

Add a pyproject.toml and define dependency versions for a stable install;

Add additional users by config file.

This enables us to automatically deploy worker processes without using the admin account.

Error when deleting an instance

I just received the following error when trying to delete an instance. Might be a migration problem.
Here is the stacktrace:

ERROR:koi_api:Exception on /api/model/0c25b10c0fad11eb9980b49691475fb4/instance/63e19d7424db11ebb4dbb49691475fb4 [DELETE]
Traceback (most recent call last):
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "d:\koi\api\venv\lib\site-packages\pymysql\cursors.py", line 148, in execute
    result = self._query(query)
  File "d:\koi\api\venv\lib\site-packages\pymysql\cursors.py", line 310, in _query
    conn.query(q)
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 775, in _read_query_result
    result.read()
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "d:\koi\api\venv\lib\site-packages\pymysql\protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "d:\koi\api\venv\lib\site-packages\pymysql\err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1054, "Unknown column 'label.label_etag' in 'field list'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "d:\koi\api\venv\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "d:\koi\api\venv\lib\site-packages\flask\app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "d:\koi\api\venv\lib\site-packages\flask_restful\__init__.py", line 468, in wrapper
    resp = resource(*args, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\flask\views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\flask_restful\__init__.py", line 583, in dispatch_request
    resp = meth(*args, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\koi_api\resources\base.py", line 100, in wrapperA
    return func(self, *args, me=me, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\koi_api\resources\base.py", line 155, in wrapperM
    return func(self, *args, me=me, model=model, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\koi_api\resources\base.py", line 187, in wrapperM
    return func(self, *args, me=me, model=model, instance=instance, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\koi_api\resources\instance.py", line 586, in delete
    db.session.delete(instance)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\scoping.py", line 163, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\session.py", line 2061, in delete
    self._delete_impl(state, instance, head=True)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\session.py", line 2088, in _delete_impl
    state.manager.mapper.cascade_iterator("delete", state)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\mapper.py", line 3104, in cascade_iterator
    halt_on,
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\relationships.py", line 1942, in cascade_iterator
    state, dict_, self.key, passive=passive
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\relationships.py", line 1921, in _value_as_iterable
    for o in impl.get_collection(state, dict_, x, passive=passive)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\dynamic.py", line 110, in get_collection
    history = self._get_collection_history(state, passive)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\dynamic.py", line 240, in _get_collection_history
    return CollectionHistory(self, state, apply_to=c)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\dynamic.py", line 405, in __init__
    self.unchanged_items = util.OrderedIdentitySet(coll)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\util\_collections.py", line 713, in __init__
    for o in iterable:
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3560, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\sql\elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1130, in _execute_clauseelement
    distilled_params,
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\util\compat.py", line 182, in raise_
    raise exception
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "d:\koi\api\venv\lib\site-packages\sqlalchemy\engine\default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "d:\koi\api\venv\lib\site-packages\pymysql\cursors.py", line 148, in execute
    result = self._query(query)
  File "d:\koi\api\venv\lib\site-packages\pymysql\cursors.py", line 310, in _query
    conn.query(q)
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 775, in _read_query_result
    result.read()
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "d:\koi\api\venv\lib\site-packages\pymysql\connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "d:\koi\api\venv\lib\site-packages\pymysql\protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "d:\koi\api\venv\lib\site-packages\pymysql\err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1054, "Unknown column 'label.label_etag' in 'field list'")
[SQL: SELECT label.label_id AS label_label_id, label.label_uuid AS label_label_uuid, label.label_last_modified AS label_label_last_modified, label.label_etag AS label_label_etag, label.label_key AS label_label_key, label.sample_id AS label_sample_id, label.file_id AS label_file_id
FROM label
WHERE %(param_1)s = label.sample_id]
[parameters: {'param_1': 1697}]
(Background on this error at: http://sqlalche.me/e/13/e3q8)

bulk-upload of samples

The system should be able to accept bulk-uploads of training samples.
This would increase the usability when using premade training sets.

instances should only be finalized when all parameters are set

missing README.md

The repository is missing a README.md.

Add a health indicator for the service

This feature can be used to check the system without authentication and could be the basis for a function yet to be implemented in koi_core to check whether an API instance is reachable;

do not forget previous instance states

When a new instance state (inference data, training data) is added, the old version is omitted.
It should be possible to go back to older versions of a specific instance. This helps with debugging and when having deployment or training issues.

reuploading of model code does not clear the previous list of parameters

Flat vs hierarchical API Design

Currently, we are mixing two separate Paradigms of API Design: We have UUIDs typically used in a flat hierarchy (e.g. /api/instance/{uuid}) but use hierarchical API paths (model/{uuid}/instance/{uuid}).

We should reconsider if we still continue to use UUIDs or if we should switch or extend to flat API paths.

Tag System

We should extend our tag system (obsolete and consumed) by user-defined tags. Therefore we should rethink the tags (including obsolete and consumed) as part of a fully customized tag system. Each sample should have a possible empty of assigned tags.

There should also be the possibility to filter the samples by one or more tags via query parameters: samples?tag=tag1,tag2,tag3 or samples?tag=tag1&tag=tag2&tag=tag3

This will DEPRECATE the properties obsolete and consumed

Descriptor files are dropped when uploading new files.

If there are any files registered for an descriptor, uploading a new file simply drops the previoud file.
We schould delete the persistent file and check if the change is propagated correctly.

Add logging using the python logging interface

OpenAPI 3.0.X Specification

Consider upgrading SQLAlchemy and FLASK

Currently, the dependencies for this project are kept at SQLAlchemy<1.4.0 and FLASK<2.0.0.