hydrospheredata / hydro-serving-sdk Goto Github PK

We have YAML resource definitions for every serving entity. Need to migrate and refactor the code from CLI to the SDK.

Need Jenkins support to run tests and release artefacts.

By the request of @BeardedWhale, an external model needs to be implemented

All contract fields names are equal to "model"

This code

cluster = Cluster("https://cluster/")
model = Model.find_by_id(c, 4)
[tensor.name for tensor in m.contract.predict.inputs]

will return

['model',
 'model',
 'model',
 'model',
 'model',
 'model',
 'model',
 'model',
 'model',
 'model',
 'model',
 'model']

instead of proper field names like ['age', 'sex', ... ]

Concept of a signature isn't preserved in the ecosystem anymore, so it's better to stick with a common terminology. Contracts are used in the serving.yaml.

Why do we need a separate with_name building block for Model?

model = (
    sdk.Model()
    .with_name('RandomForest')
)

Creating Monitoring is performed differently.

monitoring = sdk.Monitoring('RandomForestMetric')

• ModelVersion
• Servable
• HostSelector
• Application

return_type argument confuses users. This argument not only specifies the return type of the request, but also the submission type.

There is no API for it, we'll implement it as a filter on top of #32

I want to attach a metric to a model, but some configurations of MetricSpecConfig raise an Exception

Steps to reproduce:

Following code

import hydrosdk
c = hydrosdk.cluster.Cluster("http://localhost")
m = hydrosdk.model.Model.find(c, "adult_classification", 1)

metric_config = hydrosdk.monitoring.MetricSpecConfig(m.id, 4.2, hydrosdk.monitoring.TresholdCmpOp.GREATER)
hydrosdk.monitoring.MetricSpec.create(c, "test_metric", m.id, metric_config)

Raises an execption

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-21-5e9ddd6c8eab> in <module>
----> 1 hydrosdk.monitoring.MetricSpec.create(c, "test_metric", m.id, metric_config)

~/anaconda3/lib/python3.7/site-packages/hydrosdk-2.2.0.dev0-py3.7.egg/hydrosdk/monitoring.py in create(cluster, name, model_version_id, config)
     66         else:
     67             raise Exception(
---> 68                 f"Failed to create a MetricSpec. Name={name}, model_version_id={model_version_id}. {resp.status_code} {resp.text}")
     69 
     70     @staticmethod

Exception: Failed to create a MetricSpec. Name=test_metric, model_version_id=2. 400 The request content was malformed:
Invalid ThresholdCmpOperator kind Gt

This error happens also with hydrosdk.monitoring.TresholdCmpOp.GREATER_EQ as an argument.

All other variations of hydrosdk.monitoring.TresholdCmpOp wor fine

Move method servable_lock_till_serving to the Servable class as other codebases utilize this method.

In model.read_yaml install_command, training-data, metadata fields are ignored

Implement the deployment mechanism for a LocalModel and Model classes.

In case of LocalModel - you have all the required metadata and payload. You can just build a docker image and run it. Dockerfile for the model image could be found here: https://github.com/Hydrospheredata/hydro-serving-manager/blob/master/src/main/scala/io/hydrosphere/serving/manager/domain/model_build/BuildScript.scala

With Model it's quite easy - the manager service already did heavy lifting for you and pushed the resulting image to registry. Pull and run.

Define models using https://pydantic-docs.helpmanual.io/

LocalModel.training-data contains path to local csv file or path to object in s3 bucket

Implement the data upload as in CLI.

Currently we have GRPC service definitions and HTTP endpoint for gateway.
GRPC operates on proto messages which are not convenient to use in Python. And JSON requires de-/serializers to use with numpy types.

We need to design an interface to seamlessly intregrate numpy/pandas data types and send to model data ingestion.

Tests use requests library to send messages to locally deployed serving cluster.
It's essential that they are mocked for instance with requests-mock.

Marked as bug because Jenkins is unable to run tests.

Need to generate appropriate mocks using numpy data types for a ModelContract and ModelSignature.

There are some tests, but they might be broken.