monte-carlo-data / monte-carlo-python-sdk-examples

Monte Carlo Python SDK Examples

These examples use Pycarlo, Monte Carlo's Python SDK and the Monte Carlo CLI.

Table of Contents

1. Utility Setup
2. Quick Start
3. Standalone Scripts

Utility Setup

Some of the scripts in this repository may be called from the main utility runner or as standalone scripts.

1. Navigate to a desired directory where the repository will reside
2. Clone or download the git repository

   git clone https://github.com/monte-carlo-data/monte-carlo-python-sdk-examples.git

3. You can choose from an existing or new virtual environment or use the base python installation as the interpreter. In either case, make sure to use python3.12 as the base interpreter
4. Install all python modules:

   python3.12 -m pip install -r requirements.txt

(back to top)

Quick Start

Use the --help/-h flag for details on the commands/utilities available.

python3.12 mcdsdksamplerunner.py -h

If the Monte Carlo CLI has not been configured before, running any utility will prompt for Monte Carlo credentials to generate new tokens. This only applies for accounts not using SSO.

Example:

(venv) python3.12 mcsdksamplerunner.py  monitors bulk-set-freshness-sensitivity -p demo -i /Users/hjarrin/Downloads/freshness_thresholds_auto.csv -w aaaa7777-7777-a7a7-a7a7a-aaaa7777

                    
            ███╗   ███╗ ██████╗ ███╗   ██╗████████╗███████╗     ██████╗ █████╗ ██████╗ ██╗      ██████╗ 
            ████╗ ████║██╔═══██╗████╗  ██║╚══██╔══╝██╔════╝    ██╔════╝██╔══██╗██╔══██╗██║     ██╔═══██╗
            ██╔████╔██║██║   ██║██╔██╗ ██║   ██║   █████╗      ██║     ███████║██████╔╝██║     ██║   ██║
            ██║╚██╔╝██║██║   ██║██║╚██╗██║   ██║   ██╔══╝      ██║     ██╔══██║██╔══██╗██║     ██║   ██║
            ██║ ╚═╝ ██║╚██████╔╝██║ ╚████║   ██║   ███████╗    ╚██████╗██║  ██║██║  ██║███████╗╚██████╔╝
            ╚═╝     ╚═╝ ╚═════╝ ╚═╝  ╚═══╝   ╚═╝   ╚══════╝     ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝ ╚═════╝ 
                                                                                                         
        
2024-08-13 16:15:28 INFO - running utility using 'demo' profile
2024-08-13 16:15:28 INFO - checking montecarlo cli version...
2024-08-13 16:15:28 INFO - montecarlo cli present
2024-08-13 16:15:28 INFO - validating montecarlo cli connection...
2024-08-13 16:15:29 ERROR - unable to validate token
2024-08-13 16:15:29 INFO - creating new token
MC Username: [email protected]
MC Password: 
2024-08-13 16:15:46 INFO - token stored successfully
2024-08-13 16:15:46 INFO - starting input file validation...
2024-08-13 16:15:46 INFO - updating freshness rules...
2024-08-13 16:15:48 INFO - freshness threshold updated successfully for table hxe:dev_schema.offer
2024-08-13 16:15:49 INFO - freshness threshold updated successfully for table hxe:dev_schema.subscription
2024-08-13 16:15:49 INFO - freshness threshold updated successfully for table hxe:dev_schema.zuora_invoice
[COMPLETE] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

Note: If your account is using SSO, generate the token manually from the UI and store them in ~/.mcd/profiles.ini or run the montecarlo configure command by following the onscreen prompts.

(back to top)

Standalone Scripts - In Progress

Very brief descriptions of standalone examples and the link to the main example/project file are provided here. For more information on an example, look at the source file.

Insights Reporting

Example Topic	Discussion
Import Insights to BigQuery	Script to import Monte Carlo's insight reports directly to BigQuery.
Import Insights to Databricks	Script to import Monte Carlo's insight reports directly to Databricks.

Tagging Assets

Example Topic	Discussion
Key Asset Tagger	A script that will take all Key Asset Importance Scores and add them as a tag.

Lineage

Example Topic	Discussion
Add External Lineage	An example class and execution to make adding an external lineage node and a set of edges to that new node.
Reports in Database Schema	A script that will provide a list of all Looker reports used by tables in a specific schema.
Get All Tables Upstream from a Report	Outputs a csv of any DWH/DL tables upstream from an inputted BI Report.
Get Recent Incidents Upstream from a Report	Prints a boolean (if there is any incidents) if there are upstream incidents and a list of affected upstream tables from an inputted BI Report.
Get Downstream Assets from an Asset	Fetches all downstream assets from a specified asset to a csv.
Get Lineage Graph for All Tables	Fetches all tables and edges, saves to csv files.