picnicml / doddle-serving Goto Github PK

This issue is a continuation of this Reddit question and serves as an initial discussion regarding design choices and collaboration on this project.

High-level requirements of the project:

• the ability to expose trained doddle-models as a REST API
• the ability to serve multiple models and multiple versions of the same model simultaneously (A/B testing different models/versions)
• a mechanism to group multiple inference examples into a batch before they are sent to the actual model (similar to the groupedWithin operator and this Apache Beam trigger)
• a CLI for model deployments

Technology Candidates

• Akka Streams
• fs2
• monix

Resources:

• Tensorflow Serving (a good example of what we are trying to port to Scala)