michalbrys / kedro-vertexai

Kedro Plugin to support running workflows on GCP Vertex AI Pipelines

Kedro Vertex AI Plugin

About

The main purpose of this plugin is to enable running kedro pipeline on Google Cloud Platform - Vertex AI Pipelines. It supports translation from Kedro pipeline DSL to kfp (pipelines SDK) and deployment to Vertex AI service with some convenient commands.

The plugin can be used together with kedro-docker to simplify preparation of docker image for pipeline execution.

Documentation

For detailed documentation refer to https://kedro-vertexai.readthedocs.io/

Usage guide

Usage: kedro vertexai [OPTIONS] COMMAND [ARGS]...

  Interact with Google Cloud Platform :: Vertex AI Pipelines

Options:
  -e, --env TEXT  Environment to use.
  -h, --help      Show this message and exit.

Commands:
  compile         Translates Kedro pipeline into JSON file with Kubeflow...
  init            Initializes configuration for the plugin
  list-pipelines  List deployed pipeline definitions
  run-once        Deploy pipeline as a single run within given experiment.
  ui              Open VertexAI Pipelines UI in new browser tab

Configuration file

kedro init generates configuration file for the plugin, but users may want to adjust it to match the run environment requirements. Check documentation for details - kedro-vertexai.readthedocs.io