dmschauer Goto Github PK

amazon-redshift-cookbook

Amazon Redshift Cookbook, Published by Packt

awesome-data-engineering

A curated list of data engineering tools for software developers

aws-cdk-lambda-pytest

AWS CDK code written in Python to create an AWS Lambda function (also in Python) that does a basic ETL transformation. The project shows how to add pytest unit tests to the Lambda function within the CDK project.

aws-cdk-save-spotify-data

Use AWS CDK (Python) to periodically call Spotify APIs and store artist data using serverless services (Lambda, API-GW, DynamoDB, S3, EventBridge)

aws-cdk-twitter-bot-thirukkuralpoem

Use AWS CDK (Python) to create a Twitter Bot that tweets a poem once a day. Uses Lambda, API-GW and DynamoDB.

aws-dva-code

Code for our AWS Certified Developer Associate course

aws-etl-orchestrator

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

aws-glue-apache-spark-unittesting-using-localstack

Covers a use-case on how to setup local AWS Glue and Apache Spark environment to perform automated unit testing using localstack. Code+Tests = examples + help on getting started with local setup. The setup script helps setup the required environment that can be used as a base to write codebase and tests required for specific app requirements.

aws-glue-cdk-cicd

Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines

aws-glue-interactive-sessions

Code for a blog article. Main reference is https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions.html

aws-glue-jobs-unit-testing

Demo code to illustrate the execution of PyTest unit test cases for AWS Glue jobs in AWS CodePipeline using AWS CodeBuild projects

aws-glue-local-dev-and-test

Code for a blog article. Mainly copied from https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python

aws-glue-local-development

aws-glue-local-mocking

Code for a blog article. Main reference is: https://medium.com/@dominikschauer/professional-aws-glue-pyspark-development-mocking-aws-services-for-unit-tests-e6c222e95933

aws-glue-monorepo-style

Example of AWS Glue Jobs and workflow deployment with terraform in monorepo style. Code here supports the miniseries of articles about AWS Glue and python.

aws-sagemaker-deployment-test

I did a simple test to see how deploying a machine learning model on AWS Sagemaker and thus turning it into an API works. Since scikit-learn models require less dependencies than e.g. TensorFlow models I went with them for this test. To do so I used a tutorial.

blackburn

A Hugo theme built using Yahoo's Pure CSS

brutalist-blog

Minimalistic Jekyll theme that automatically supports dark and light modes for blogs

complete-dbt-bootcamp-zero-to-hero

Supplementary Materials for the dbtlearn.com Udemy course

contrast

Jekyll Theme

dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files

data-engineering-project-template

This is a template you can use for your next data engineering portfolio project.

data-engineering-with-apache-spark-delta-lake-and-lakehouse

Data Engineering with Spark and Delta Lake

data-engineering-with-aws

Data Engineering with AWS, Published by Packt

data-engineering-with-databricks-english

data-engineering-with-python

Data Engineering with Python, published by Packt

data-pipeline-with-cdk

data-pipelines-with-apache-airflow

Code for Data Pipelines with Apache Airflow

dataops-platform-airflow-dbt

Build DataOps platform with Apache Airflow and dbt on AWS

datapipeline