annawykes / devops-for-databricks

devops-for-databricks

devops-for-databricks

IaC folder contains examples of IaC for Databricks using Terraform and Pulumi

To run Databricks REST API the Demo

1. Create a project in Azure DevOps called: devops-for-databricks
2. Create an Empty Azure DevOps Repo as clone of this Github repo
3. Create an Azure KeyVault with the variables stated below
4. Create a Variable Group in Pipeline->Library called: devops-for-dbx-vg
5. Connect the Variable Group to the Azure Keyvault and add all the variables
6. Create an Azure Artifact feed called: devops-for-databricks
7. Run the Pipeline

The Pipeline will need an Azure Secret Vault with the following secrets.

DBXInstance: Databricks instance, eg: adb-631237481529976.16
ResourceGroup: Resource Group where Databricks instance is
SubscriptionID: Subscription ID where everything runs in Azure
SVCApplicationID: Application (client) ID for the Service Principal
SVCDirectoryID: Directory (tenant) ID for the Service Principal
SVCSecretKey: Secret value for the Service Principal
WorkspaceName: Name of the Databricks Workspace

Pipeline Run Example