dominodatalab / domino-research

Projects developed by Domino's R&D team

Domino Research

This repo contains projects under active development by the Domino R&D team. We build tools that help Data Scientists and ML engineers train and deploy ML models.

Active Projects

Here’s what we’re working on:

• 🌉 Bridge - deploy directly from your registry, turning it into a declarative source of truth for your model hosting.

• 🛂 Checkpoint - adds 'Pull Requests' to your registry to create a better process for promoting models to production.

• 🎇 Flare - monitor models and get alerts without capturing, storing or processing production inference data.

domino-research's Issues

Add created at and approved at to the PR model

Create IAM for Bridge run

Use bridge init to create an IAM user that Bridge uses at runtime. This would simplify the user configuration.

Add approver as a tag to model version

Upon approval, add the approver to the model version as a tag

Gracefully handle moves to 'none/archived' stage

• Do not intercept None -> archived
• Return no champion for moved to archived/none

Add readme

Move to postgres

Optimize endpoint create/update times

Endpoints start very slowly. Consider the following optimizations:

• Larger default instances to run conda install faster
• Use a requirments.txt file instead of a conda yaml in model packaging
• Use a multi-model endpoint that doesn't require new instances/restarts to update
• Use a custom image to achieve smaller image size

Broken tab selection in the MLflow nav

Add better logging to sagemaker deploy target

• Add info level logging around all major API calls

Drop polling time to 1000

Add MLFLow stand up to quickstart

Add to the quickstart with a guide on how to stand up MLFlow if users do not have an instance

Change comment empty state

"No comment provided"

Save champion version at close time

Handle boto config when mlflow isn't using s3 backend and needs different creds

If the user uses MLflow with a non-s3 backend for boto, then we need a way to provide two different configs for mlflow and our use of boto. MLFlow will always read env, so we need to allow our aws config to be overriden and NOT look at the same place as the mlflow boto

PR count in MLFlow nav is all not open

Remove references to local filesystem from tests

Validate steps in local MLFlow guide do not override Bridge credentials

Query S3_ENDPOINT from MLflow registry to simplify config

See if we can pull the S3 endpoint that the registry is using so that we can avoid duplicating this config across the registry and Bridge itself when we use a non-s3 backend

Add version update check on boot/run

On run, check for more recent release on GitHub and prompt user to update

Teardown/destroy is incomplete

• Block until all endpoints are deletable to avoid leaving endpoints
• Empty all artifacts from bucket so that it can be deleted
• Separate the model-level and infra-level teardown for use in tests

Create README/quickstart

Report deployment URL as version tag

Update model version with info on where it is deployed and how it can be queries

500s during model update in localhost target

This looks like the error

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
    return self.finalize_request(rv)
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 1535, in finalize_request
    response = self.make_response(rv)
  File "/opt/conda/lib/python3.9/site-packages/flask/app.py", line 1727, in make_response
    raise TypeError(
TypeError: The view function did not return a valid response. The return type must be a string, dict, tuple, Response instance, or WSGI callable, but it was a int.

Deploy to Heroku button for Checkpoint

This could be tricky given our use of Python and React (requiring two buildpacks). They have a Dockerfile option, but the start up time would be 5+ minutes. Is there an easier way for us to do this?

Remove our MLFlow as default in checkpoint

Handle a NOOP PR (version replacing itself)

Add id/number to PR list view

Investigate oauth/authn/authz

Oauth proxy?

Handle race conditions

NOOP race condition

• PR opens to move version to archive
• Version moves to archive by another means (manually or via another PR)
• Original PR errors out when approving

Target Version race condition

• You are targeting some change to production.
• Someone else makes a change to production.
• your diff is now out of date and you're an idiot

Reflect Endpoint State in UI (two-way sync)

Show status of endpoints (models/versions) to users in the mlflow UI. Consider:

• Comments
• Tags

Work out what needs provisioning once and build a provisioning mechanism

• S3 bucket for artifacts
• SageMaker exec role

Manually sync a single model

Allow user to manually sync a single model from the registry to sagemaker

Assert the version of Python interpreter

I accidentally used an older version of Python (3.8), and this caused the following exception in bridge run:

AttributeError: 'str' object has no attribute 'removesuffix'

The function in question was, indeed, introduced in Python 3.9.

I will add an assertion in the beginning of the call stack to guarantee that the interpreter version is kosher (>=3.9).

Command line to tear down

UTC times to current timezone

Model update/synchronization bug

Replication:

1. Add a couple model versions to registry
2. Init bridge in a fresh account
3. Bridge creates an endpoint/model
4. Before bridge is done creating the endpoint, add and tag a new version destined for the endpoint
5. Bridge skips the update.
6. After some time (perhaps when endpoint becomes updateable) bridge attempts update and then fails due to existing model.

Back to registry should redirect to /models

Assisted construction of endpoint invocation command

User should be able to close own PR but not approve

depends on #77
depends on #61

Add a default title template to new PR form

"Promote {model name} version {v_num} to {target_stage}"

Handle non-standard starting states for create/update

Query ALL versions in a stage in MLFlow

Our logic currently only fetches the most recent version in each stage.

Consider fetching all the versions in a stage so that we can enable A/B testing by deploying N versions to the endpoint for a stage.

Think about updating quickstart for Windows.

cc @JoshBroomberg

Make repo look real

Fetch author and reviewer username from header

depends on #77

Fetch from header or default to sentinal "anonymous"

Better current_stage handling

• remove current_stage from create time and from DB

In the view one PR endpoint

• Query current stage and add it
• Query the current version in the target stage
• Query all the other info we need for the view

Add precommit hook to repo for the Bridge project

• black, then flake8, then mypy

Move mlflow image to quay.io

Optional per-registry name prefix

If you have more than one MLFlow registry and attempt to run a bridge worker pointing at each, and deploying to the same AWS account/region then the models from each registry may clobber each other.

Solution: add a per-registry prefix to AWS resources so that multiple registries can cohabit the same account and region.

Change default template title

Remove 'Promote' so that it makes sense with Staging -> Archive/None