zkan / airflow-tutorial-1

🐍💨 Airflow tutorial for PyCon 2019

Airflow Tutorial

This repo contains the materials for the pipelines tutorial on Pycon -> from scripts soups to Airflow.

The tutorial covers:

• Setting up local databases
• Creating basic ETL pipelines in Python: query APIs, load data to databases, perform data cleaning and filtering and persist the consumption ready data
• How to set a local instance of Airflow and get it running
• Creating basic DAGS in Airflow
• Transform script soups ETLS into Airflow dags
• Set up an Airflow instance in Azure

To add:

• Setting a Kubernetes powered instance on Azure AKS
• Adding CI/CD to using Azure pipelines

If you are interested in following along visit: https://airflow-tutorial.readthedocs.io/en/latest/

The setup instructions can be found at: https://airflow-tutorial.readthedocs.io/en/latest/setup.html

If you would like to experiment with Azure follow this link to get a free trial subscription with 150 dollars.

🚀 PRs and Issues are welcome

License

This repo is licensed using a CC-BY so you are free to use, remix, and share so long attribution is provided to the original author.