datatalksclub / data-engineering-zoomcamp Goto Github PK

View Code? Open in Web Editor NEW

22.5K 406.0 4.8K 6.36 MB

Free Data Engineering course!

HCL 0.02% Dockerfile 0.10% Python 0.66% Shell 0.04% Jupyter Notebook 98.50% Java 0.67% PowerShell 0.01%

data-engineering kafka spark dbt docker prefect

data-engineering-zoomcamp's Introduction

Data Engineering Zoomcamp

Register in DataTalks.Club's Slack
Join the #course-data-engineering channel
Join the course Telegram channel with announcements
The videos are published on DataTalks.Club's YouTube channel in the course playlist
Frequently asked technical questions

Syllabus

Module 1: Containerization and Infrastructure as Code
Module 2: Workflow Orchestration
Workshop 1: Data Ingestion
Module 3: Data Warehouse
Module 4: Analytics Engineering
Module 5: Batch processing
Module 6: Streaming
Workshop 2: Stream Processing with SQL
Project

Taking the course

2024 Cohort

Start: 15 January 2024 (Monday) at 17:00 CET
Registration link: https://airtable.com/shr6oVXeQvSI5HuWD
Cohort folder with homeworks and deadlines
Launch stream with course overview

Self-paced mode

All the materials of the course are freely available, so that you can take the course at your own pace

Follow the suggested syllabus (see below) week by week
You don't need to fill in the registration form. Just start watching the videos and join Slack
Check FAQ if you have problems
If you can't find a solution to your problem in FAQ, ask for help in Slack

Syllabus

Note: NYC TLC changed the format of the data we use to parquet. In the course we still use the CSV files accessible here.

Module 1: Containerization and Infrastructure as Code

Course overview
Introduction to GCP
Docker and docker-compose
Running Postgres locally with Docker
Setting up infrastructure on GCP with Terraform
Preparing the environment for the course
Homework

More details

Module 2: Workflow Orchestration

Data Lake
Workflow orchestration
Workflow orchestration with Mage
Homework

More details

Workshop 1: Data Ingestion

Reading from apis
Building scalable pipelines
Normalising data
Incremental loading
Homework

More details

Module 3: Data Warehouse

Data Warehouse
BigQuery
Partitioning and clustering
BigQuery best practices
Internals of BigQuery
BigQuery Machine Learning

More details

Module 4: Analytics engineering

Basics of analytics engineering
dbt (data build tool)
BigQuery and dbt
Postgres and dbt
dbt models
Testing and documenting
Deployment to the cloud and locally
Visualizing the data with google data studio and metabase

More details

Module 5: Batch processing

Batch processing
What is Spark
Spark Dataframes
Spark SQL
Internals: GroupBy and joins

More details

Module 6: Streaming

Introduction to Kafka
Schemas (avro)
Kafka Streams
Kafka Connect and KSQL

More details

Workshop 2: Stream Processing with SQL

More details

Project

Putting everything we learned to practice

Week 1 and 2: working on your project
Week 3: reviewing your peers

More details

Overview

Prerequisites

To get the most out of this course, you should feel comfortable with coding and command line and know the basics of SQL. Prior experience with Python will be helpful, but you can pick Python relatively fast if you have experience with other programming languages.

Prior experience with data engineering is not required.

Instructors

Past instructors:

Asking for help in Slack

The best way to get support is to use DataTalks.Club's Slack. Join the #course-data-engineering channel.

To make discussions in Slack more organized:

Follow these recommendations when asking for help
Read the DataTalks.Club community guidelines

Supporters and partners

Thanks to the course sponsors for making it possible to run this course

Do you want to support our course and our community? Please reach out to [email protected]

Star History

data-engineering-zoomcamp's People

Contributors

Stargazers

Watchers

Forkers

rohitpandey13 victoriapm gvinoth7 paty-oliveira ldebb archie17 christiankandinata emilien-mipt unclemokus datalearns hparmar12 adagloria agidee-o alomdaelmasry hailu21 matyi101 damu1995 admariner mragungsetiaji trungnghiahoang96 praveenanantharaman abiradar itsmuriuki tuantmb rancychepchirchir girijeshcse becho rohanverma94 rahulsaini irfansofyana hirajanwin iuriimattos2 pursh2002 dardeery faqihhamami arzucubukcu mpofukelvintafadzwa annajiat ahmadabdelhameed hernanlira bnc2050 unachka nanaakwasiabayieboateng venugpl1994 wikankun shiroiusagi4486 jachur ameer191815 blueband lozanocelia vaibhav65 nurtantioquidar turanlioglu nmbazima egorwk4 phathumandanedesigns efecaliskan therockstardba balakrishnanaidu erbilensaid eroyzman mehrozkhan hodlin jasonantonio10 shankarpd shobanac algonacci kabirodavies tinyfire973 flowervine shirishkz laislimadev devcsar yaretzyc aditya-zutshi harizjet hashphoenix sunilbhatia sejalv cmeninwa leboharmony balchwhit itsmevikrant manianls shaunstanislauslau martinsedm cosmiccomposer alchemist1218 lennart-damen isavinit home-of-forked-repos ekeminiumanah mkt-ds muizadekunle mattchurgin mutum amart85 shadah-17 erni1985 kuldeep27396

data-engineering-zoomcamp's Issues

Timecodes for "DE Zoomcamp 5.3.2 - Spark DataFrames"

Youtube video: https://www.youtube.com/watch?v=ti3aC1m3rE8

Timecodes for "DE Zoomcamp 5.3.3 - (Optional) Preparing Yellow and Green Taxi Data"

Youtube video: https://www.youtube.com/watch?v=CI3P4tAtru4

Timecodes for "DE Zoomcamp 4.5.1 - Visualising the data with Google Data Studio (Alternative A)"

Youtube video: https://www.youtube.com/watch?v=39nLTs74A3E

Timecodes for "DE Zoomcamp 5.1.2 - Introduction to Spark"

Youtube video: https://www.youtube.com/watch?v=FhaqbEOuQ8U

Timecodes for "DE Zoomcamp 5.4.3 - Joins in Spark"

Youtube video: https://www.youtube.com/watch?v=lu7TrqAWuH4

Timecodes for "DE Zoomcamp 3.3.2 - BigQuery Machine Learning Deployment"

Youtube video: https://www.youtube.com/watch?v=BjARzEWaznU

Timecodes for "DE Zoomcamp 5.6.3 - Setting up a Dataproc Cluster"

Youtube video: https://www.youtube.com/watch?v=osAiAYahvh8

Timecodes for "DE Zoomcamp 5.3.4 - SQL with Spark"

Youtube video: https://www.youtube.com/watch?v=uAlp2VuZZPY

Week 4 - DBT build ARGs not being correctly passed after FROM

Hey DataTalks zoompcamp team!

I'm not sure if I'm the only person having this issue but it looks like the Dockerfile file provided in Week 4 for building dbt-bigquery fails to build. Namely the ARGs defined at the beginning of the Dockerfile aren't made available during each part of the multi-stage build.

I think each particular ARG needs to be defined after each FROM statement for it to be accessible during a particular build stage.

i.e:

...
FROM base as dbt-core
ARG [email protected]
RUN echo python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" \
  && python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core"

The Docker Docs on this behaviour make a claim that you don't need to re-define a default value after FROM (i.e leave it as ARG dbt_core_ref but that hasn't worked on my system using Docker 20.10.12.

However, I can still run this dag via the Airflow CLI. I'm not sure what's the problem here - I built the image on two different machines, one worked fine, and one got stuck like this.

(New-Object Net.WebClient).DownloadFile("https://dl.google.com/dl/cloudsdk/channels/rapid/GoogleCloudSDKInstaller.exe", "$env:Temp\GoogleCloudSDKInstaller.exe")

& $env:Temp\GoogleCloudSDKInstaller.exe

datatalksclub / data-engineering-zoomcamp Goto Github PK

data-engineering-zoomcamp's Introduction

Data Engineering Zoomcamp

Taking the course

2024 Cohort

Self-paced mode

Syllabus

Overview

Prerequisites

Instructors

Asking for help in Slack

Supporters and partners

Star History

data-engineering-zoomcamp's People

Contributors

Stargazers

Watchers

Forkers

data-engineering-zoomcamp's Issues

Recommend Projects

Recommend Topics

Recommend Org