[question] about mist

[spushkarev]

Hi @geoHeil !
Thanks for the question. To list a few key differences.

1. Fundamentally job server is designed to create jobs to be run by big data engineers. Hydrosphere Mist provides higher level decoupling for web developers.
   Job server API:
   POST /jars/app_name
   POST /contexts/context_for_myapp
   POST /data/params_for_my_job
   POST /jobs/my_job

Versus Mist API for the same action:
POST /weather-forecast
{
lat: “37.777114”,
long: “-122.419631”
radius: 100
}

2. Mist provides reactive and true asynchronous API - i.e. you could run streaming job and subscribe for updates. Also you'll be able to run parametrized streaming job or modify a running job to receive new results.

3. Synchronous realtime API for high throughput - we are working on adding a model serving support for online queries with low latency.

4. Python jobs support - you could implement Spark programs in Python to be executed from Mist.

5. Spark contexts isolation and resource management on driver nodes.

6. And others

Take a look at the blog post for some background details
https://medium.com/@stepanpushkarev/architecting-smart-applications-on-top-of-apache-spark-b0fcab6ea400

pipeline.io is not a product - it's just a demo for training classes with all the buzzwords inside a single docker container :) While it is cool I'm not sure how to use it.

seldom is is a platform with all the dependencies. Also I do not see big data technologies there.

Mist is strongly focused on providing model serving and data pipeline API on top of Spark.
Regarding testing, deployment and monitoring - we are working on other independent micro products which will complement Mist in a long term.

[geoHeil]

Sounds pretty awesome.

Synchronous realtime API for high throughput - we are working on adding a model serving support for online queries with low latency.
When do you plan tho have low latency serving up and running?

[spushkarev]

Its a tough problem and it depends on actual implementation of the serving action. The goal is to re-use existing Spark code and not to export the model to PMML->NoSQL DB->custom online serving layer.
We hope to have the 1st version ready at January.
If you are interested we'll share a design vision and then will collaborate on implementation.

[geoHeil]

I am interested but not sure if I will really be able to support you. This
will depend on if we will use mist internally.
spushkarev [email protected] schrieb am Di. 22. Nov. 2016 um 23:53:

Its a tough problem and it depends on actual implementation of the serving
action. The goal is to re-use existing Spark code and not to export the
model to PMML->NoSQL DB->custom online serving layer.
We hope to have the 1st version ready at January.
If you are interested we'll share a design vision and then will
collaborate on implementation.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#93 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnc9Krt8YMBKa0K8xJMGh3kojq70x9Aks5rA3J2gaJpZM4K5me6
.

[geoHeil]

Do I understand that for your solution containers (docker) are mandatory? Or would it work in a yarn only world as well?

[spushkarev]

Docker is not mandatory, Mist is Scala-Akka application and could be run anywhere.