This repo is my playground to try out various data engineering stuff. The used services/tools/design is not always the best choice or sometimes unnecessary cumbersome โ this just reflects me trying to explore different things. At the moment, the pipeline processes Covid-19 data as follows: All infrastructure is templated in AWS CloudFormation or AWS CDK. All steps feature an alarm on failure. The stack can be deployed via Github Actions. I use poetry to manage the dependencies/virtual environment.
moritzkoerber / covid-19-data-engineering-pipeline Goto Github PK
View Code? Open in Web Editor NEWA Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation and CDK, deployable via Github Actions.
License: MIT License