aporia-ai / inferencedb Goto Github PK

🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)

License: Other

Dockerfile 0.86% Makefile 4.57% Smarty 2.95% Python 91.62%

machine-learning model-serving kafka s3 model-monitoring mlops

inferencedb's Introduction

InferenceDB makes it easy to stream inferences of real-time ML models in production to a data lake, based on Kafka. This data can later be used for model retraining, data drift monitoring, performance degradation detection, AI incident investigation and more.

Quickstart

Features

Cloud Native - Runs on top of Kubernetes and supports any cloud infrastructure
Model Serving Integrations - Connects to ML model serving tools like KServe
Extensible - Add your own model serving frameworks and database destinations
Horizontally Scalable - Add more workers to support more models and more traffic
Python Ecosystem - Written in Python using Faust, so you can add your own data transformations using Numpy, Pandas, etc.

Made with ❤️ by Aporia

WARNING: InferenceDB is still experimental, use at your own risk! 💀

Installation

The only requirement to InferenceDB is a Kafka cluster, with Schema Registry and Kafka Connect.

To install InferenceDB using Helm, run:

helm install inferencedb inferencedb/inferencedb -n inferencedb --create-namespace \
  --set kafka.broker=kafka:9092 \
  --set kafka.schemaRegistryUrl=http://schema-registry:8081 \
  --set kafka.connectUrl=http://kafka-connect:8083

Usage

To start logging your model inferences, create an InferenceLogger Kubernetes resource. This is a Kubernetes Custom Resource that is defined and controlled by InferenceDB.

Example:

apiVersion: inferencedb.aporia.com/v1alpha1
kind: InferenceLogger
metadata:
  name: my-model-inference-logger
  namespace: default
spec:
  topic: my-model
  events:
    type: kserve
    config: {}
  destination:
    type: confluent-s3
    config:
      url: s3://my-bucket/inferencedb
      format: parquet
      awsRegion: us-east-2

This InferenceLogger will watch the my-model Kafka topic for events in KServe format, and log them to a Parquet file on S3. See the KServe quickstart guide for more details.

Development

InferenceDB dev is done using Skaffold.

Make sure you have a Kubernetes cluster with Kafka installed (can be local or remote), and edit skaffold.yaml with the correct Kafka URLs and Docker image registry (for local, just use local/inferencedb).

To start development, run:

skaffold dev --trigger=manual

This will build the Docker image, push it to the Docker registry you provided, and install the Helm chart on the cluster. Now, you can make changes to the code, click "Enter" on the Skaffold CLI and that would update the cluster.