m-lemaire / tp

Practices - Data engineering

Tools you need

Have a stackoverflow account : https://stackoverflow.com/

Have a github account : https://github.com/

A Scala IDE :

1. Intellij Idea: https://www.jetbrains.com/fr-fr/idea/download
2. Scala plugin : https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html

Docker and Compose

Take time to read and install

https://docs.docker.com/get-started/overview/

docker --version
Docker version 20.10.14 (example)

https://docs.docker.com/compose/

docker-compose --version
docker-compose version 1.29.2 (example)

Fork OR update the repo on your own Github account

Fork

• Click on https://github.com/polomarcus/tp/fork

Update your fork

• https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

TP - Apache Kafka - Scala Client - Condutkor

• Open 'tp-docker-api

TP - Functional programming for data engineering

• Open tp-functional-programming-scala

TP - Functional programming for data engineering

• Open tp-data-processing-framework

TP - Batch with Spark and BI with metabase

• Open tp-bi-spark-framework

TP - Kafka Streams to read and write to Kafka

• https://kafka.apache.org/documentation/streams/