testnation1 / azdo-databricks

End-to-end Azure Databricks Workspace automation with Azure Pipelines

azdo-databricks

---

This repository contains examples of Azure DevOps ( azdo) Pipelines that demonstrate how an end-to-end Azure Databricks workspace automation could be done.

The Azure Pipelines can use either Terraform or scripts (with Azure CLI and ARM templates).

The main goal is to have a Databricks Delta pipeline orchestrated with Azure Data Factory while starting with an empty Azure account (with only an empty Subscription and DevOps Organization).

• all of this by simply running ./run_all.sh:

Quick start

1. Create the Subscription and DevOps Organization. If using the free tier, request a free Azure DevOps Parallelism grant by filling out the following form: https://aka.ms/azpipelines-parallelism-request

2. Fork this GitHub repository as Azure DevOps would need access to it and changing the Azure Pipelines variables would require committing and pushing changes.

3. Customize the variables:

   • admin setup variables: edit the admin/vars.sh file (especially SUFFIX and AZURE_DEVOPS_GITHUB_REPO_URL)
   • Azure Pipelines variables: edit the pipelines/vars.yml file, commit and push changes

4. Use the run_all.sh script:

export USE_TERRAFORM="yes"
export AZDO_ORG_SERVICE_URL="https://dev.azure.com/myorg/"    # or set it in vars.sh
export AZDO_PERSONAL_ACCESS_TOKEN="xvwepmf76..."              # not required if USE_TERRAFORM="no"
export AZDO_GITHUB_SERVICE_CONNECTION_PAT="ghp_9xSDnG..."     # GitHub PAT

./run_all.sh

Main steps

Security was a central part in designing the main steps of this example and reflected in the minimum user privileges required for each step:

• step 1: administrator user (Owner of Subscription and Global administrator of the AD)
• step 2: infra service principal that is Owner of the project Resources Group
• step 3: data service principal that can deploy and run a Data Factory pipeline

Step 1: Azure core infrastructure (admin setup)

Builds the Azure core infrastructure (using a privileged user / Administrator):

• this is the foundation for the next step: Resource Groups, Azure DevOps Project and Pipelines, Service Principals, Project group and role assignments.
• the user creating these resources needs to be Owner of Subscription and Global administrator of the Active Directory tenant.
• it can be seen as deploying an empty shell for a project or business unit including the Service Principal ( the Infra SP) assigned to that project that would have control over the project resources.

To run this step use one of the scripts depending on the tool preference:

• Terraform: ./admin/setup-with-terraform.sh (code)
• Scripts with Azure CLI: ./admin/setup-with-azure-cli.sh (code)

Before using either, check and personalize the variables under the admin/vars.sh file.

Step 2.1: Azure infrastructure for the data pipeline and project

Builds the Azure infrastructure for the data pipeline and project (using the project specific Infra SP):

• this is the Azure infrastructure required to run a Databricks data pipeline, including Data Lake Gen 2 account and containers, Azure Data Factory, Azure Databricks workspace and Azure permissions.
• the service principal creating these resources is the Infra SP deployed at step 1 ( Resource Group owner).
• it is run as the first stage in the Azure DevOps infra pipeline with the pipeline name defined in the AZURE_DEVOPS_INFRA_PIPELINE_NAME variable.

• there are two Azure Pipelines yaml definitions for this deployment and either one can be used depending on the tool preference:

  Terraform: pipelines/azure-pipelines-infra-with-terraform.yml (code)
  • ARM templates and Azure CLI: pipelines/azure-pipelines-infra-with-azure-cli.yml (code)

To run this step:

• either use the az cli command like run_all.sh does it.
• or use the Azure DevOps portal by clicking the Run pipeline button on the pipeline with the name defined in the AZURE_DEVOPS_INFRA_PIPELINE_NAME variable.

Before using either, check and personalize the variables under the pipelines/vars.yml file ( don't forget to push any changes to Git before running).

Step 2.2: Databricks workspace bootstrap

This step is executed together with the one above and after deploying the Azure infrastructure and the Databricks workspace itself:

• it bootstraps the Databricks workspace with the required workspace objects for a new project and pipeline, including Instance Pools, Clusters, Policies, Notebooks, Groups and Service Principals.
• the service principal creating these resources is the Infra SP deployed at step 1 and is already a Databricks workspace admin since it deployed the workspace.
• it is run as the second stage in the Azure DevOps infra pipeline with the pipeline name defined in the AZURE_DEVOPS_INFRA_PIPELINE_NAME variable.

This is run together with previous step but if it is needed to run it separately:

• in the Azure DevOps portal, before clicking the Run pipeline button on the Infra pipeline, deselect the Deploy infrastructure job.
• with Terraform, use the terraform/deployments/run-deployment.sh script file.

Step 3: Azure Data Factory data pipeline with Databricks

This step is executed after the infrastructure deployment and workspace bootstrap:

• it's a simple Azure DevOps Pipeline that deploys with ARM templates an Azure Data Factory data pipeline together with the Databricks linked service.
• it then invokes the Azure Data Factory data pipeline with the Azure DevOps Pipeline parameters.
• the service principal deploying and running the pipeline is the Data SP deployed at step 1 and it has the necessary Databricks and Data Factory permissions given at step 2.
• this service principal also has the permission to write data into the Data Lake.
• the Databricks linked service can be of two types:
  • using the Data Factory Managed Identity to authenticate to Databricks: pipelines/azure-pipelines-data-factory-msi.yml (code)
  • using an AAD Access Token of the Data SP to authenticate to Databricks: pipelines/azure-pipelines-data-factory-accesstoken.yml (code)

To run this step:

• either use the az cli command like run_all.sh does it.
• or use the Azure DevOps portal by clicking the Run pipeline button on the pipeline with the name defined in the AZURE_DEVOPS_DATA_PIPELINE_NAME variable.

It will use some of the variables under the pipelines/vars.yml file and it can be customized using pipeline parameters like database and table names, source data location, etc.