criteo / mlflow-yarn

Backend implementation for running MLFlow projects on Hadoop/YARN.

mlflow-yarn

Backend implementation for running MLFlow projects against a Hadoop/YARN backend).

To activate just do 'pip install mlflow-yarn' along your mlflow environment. It will register the plugin as an entrypoint with the YARN backend.

$ pip install mlflow-yarn

mlflow-yarn only supports Python ≥3.6.

Developement

Install from source

$ git clone https://github.com/criteo/cluster-pack
$ cd cluster-pack
$ pip install .

Examples

Example with pip only project

• Dependencies are pulled from requirements.txt

$ git clone https://github.com/criteo/mlflow-yarn
$ pip install mlflow
mlflow run tests/resources/pip_project -e compute_intersection -P size=10000 --backend yarn

Example with conda project

• Dependencies are pulled from conda.yaml

$ git clone https://github.com/criteo/mlflow-yarn
$ pip install mlflow
$ mlflow run tests/resources/conda_project -e compute_intersection -P size=10000 --backend yarn

More infos on setting up a project file.

Docker container environment is currently not supported.

mlflow-yarn's Issues

Use json schema to validate the config

Currently the parameters of backend_config.json are passed as is to skein_launcher