hagarciag / aws-concurrent-data-orchestration-pipeline-emr-livy

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

AWS Concurrent Data Orchestration Pipeline EMR Livy

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.

Description of the project folders

cloudformation

This folder contains the cloudformation template that spins up the Airflow infrastructure.

dags/airflowlib

This folder contains reusable code for Amazon EMR and Apache Livy.

dags/transform

This folder contains sample transformation scala code which transforms the movielens data files from csv to parquet.

dags/movielens_dag.py

This script contains the code for the DAG definition. It basically defines the Airflow pipeline.

License

This library is licensed under the Apache 2.0 License.