Support for Kubernetes (with Argo) about metaflow

[alexec]

Argo team would be happy to help make this a reality.

[savingoyal]

#992 has been merged. The next Metaflow release will address this issue.

[savingoyal]

#992 is now available for review

[repocho]

Ohhh my good. Thanks for this collaboration @alexec and @savingoyal !!!

[valayDave]

@nlaille : I was recently exploring building plugin support for Argo but the DAG templating seemed a little complicated to fit into Metaflow within a short amount of time.

So meanwhile I have built a plugin to support Kubernetes with isolated self packaged deployments of the runtime itself to the cluster. It will require a service-based metadata provider. It will deploy the entire workflow's runtime as a job that will orchestrate different containerized jobs. So deploying your entire flow with its container orchestration on Kubernetes becomes one command: python myflow.py --with kube:cpu=2,memory=2000,image=python:3.7 kube-deploy run --custom_param_1 hello --custom_param_2 100

It works using the @kube decorator. Supports the same features as the batch plugin. You can even use the --with kube:cpu=2,memory=2000,image=tensorflow/tensorflow:latest-py3.

https://github.com/valayDave/metaflow/tree/kube_cpu_stable

Documentation is in the links within that repo. The plugin is under constant development so there will be more features soon and hopefully even support Argo. But it is currently tied to S3 as its datastore and doesn't support GPUs.

Hope this helps!

[terrytangyuan]

@savingoyal Looks like the current support for K8s is added via K8s Python API and noticed the following the doc:

Scheduling Metaflow flows on Argo
This functionality is in-flight!

How's the integration with Argo Workflows coming along?

[terrytangyuan]

Amazing work! Looking forward to the new release!

[savingoyal]

Support for Kubernetes and Argo Workflows is now available in Metaflow 2.6.0 - https://blog.argoproj.io/human-centric-data-science-on-kubernetes-with-metaflow-7f60aad34cba

[alexec]

This is an extremely popular issue. It'd be great to see it implemented.

[alexec]

Yay!

[savingoyal]

@alexec That would be wonderful! Let me email you to set up some time.

[terrytangyuan]

Is there any update on this? BTW, the URL for Argo Workflows has changed and the old URL does not redirect correctly. Here's the new location: https://github.com/argoproj/argo-workflows

[repocho]

Thank you for making this possible !! 🥇

[repocho]

Is there a plan to include it in Metaflow official repository?

[alexec]

You can book 30m: https://calendly.com/argoproj/30m

[savingoyal]

If you would like to try out and give feedback on our Kubernetes integration, please reach out at http://slack.outerbounds.co

[terrytangyuan]

FYI we now have a low-level Python SDK officially maintained by the Argo Workflows team. It should be much easier to use it to integrate Metaflow with Argo Workflows. https://github.com/argoproj/argo-workflows/tree/master/sdks/python