josephmachado / beginner_de_project_stream

Simple stream processing pipeline

Hi 👋

I write about data engineering | data modeling | software engineering | system design

• 🔭 I write data engineering articles on Start data engineering

• 👨‍💻 All of my projects are available at https://github.com/josephmachado

• 📝 I regularly write articles on https://www.startdataengineering.com/

• 💬 Ask me about Data engineering, SQL, Databases, Data pipelines, Data infrastructure

• 📫 How to reach me https://www.startdataengineering.com/contact

• 📄 Know about my experiences https://www.linkedin.com/in/josephmachado1991/

Connect with me:

Languages and Tools:

beginner_de_project_stream's Issues

Error when using docker compose

Hi! Your blog is really cool ^^ I was trying to follow along with the stream project to learn but it didn't work for me, maybe I did something wrong?

docker-compose up -d
[+] Running 0/7

• postgres Pulling 1.1s
• serverloggenerator Pulling 1.1s
• frauddetection Pulling 1.1s
• zookeeper Pulling 1.1s
• taskmanager Pulling 1.1s
• kafka Pulling 1.1s
• jobmanager Pulling 1.1s
  [+] Running 0/190 Pulling fs layer 0.0s
• postgres Pulling 1.2s
  • 54fec2fa59d0 Waiting 0.1s
  • 30a95add0890 Waiting 0.1s
  • 57bc798d3c84 Waiting 0.1s
  • a41bedb2c5de Waiting 0.1s
  • 589548c3abb4 Waiting 0.1s
  • c4c6b75d5deb Waiting 0.1s
  • 8f8c045a6a99 Waiting 0.1s
  • 69f9dd86b24d Waiting 0.1s
  • 45bbaba740ff Pulling fs layer 0.1s
  • 1761ca7befa0 Waiting 0.1s
  • 57feb34018f4 Pulling fs layer 0.1s
  • bede8373accc Waiting 0.1s
  • 6e4c69fbe63b Waiting 0.1s
  • 8a7949704ab2 Waiting 0.1s
• serverloggenerator Pulling 1.2s
• frauddetection Pulling 1.2s
• zookeeper Pulling 1.2s
• taskmanager Pulling 1.2s
  • e9afc4f90ab0 Pulling fs layer 0.1s
  • 989e6b19a265 Pulling fs layer 0.1s
  • af14b6c2f878 Pulling fs layer 0.1s
  • 68a79816c3e1 Waiting 0.1s
  • 037cc5cb1b83 Waiting 0.1s
  • d3efdb331614 Waiting 0.1s
  • bf82d2b871ad Waiting 0.1s
  • 4ff2e8c5d83f Waiting 0.1s
    [+] Running 0/390 Waiting 0.1s
• postgres Pulling 1.3s
  • 54fec2fa59d0 Waiting 0.2s
  • 30a95add0890 Waiting 0.2s
  • 57bc798d3c84 Waiting 0.2s
  • a41bedb2c5de Waiting 0.2s
  • 589548c3abb4 Waiting 0.2s
  • c4c6b75d5deb Waiting 0.2s
  • 8f8c045a6a99 Waiting 0.2s
  • 69f9dd86b24d Waiting 0.2s
  • 45bbaba740ff Pulling fs layer 0.2s
  • 1761ca7befa0 Waiting 0.2s
  • 57feb34018f4 Pulling fs layer 0.2s
  • bede8373accc Waiting 0.2s
  • 6e4c69fbe63b Waiting 0.2s
  • 8a7949704ab2 Waiting 0.2s
• serverloggenerator Pulling 1.3s
• frauddetection Pulling 1.3s
• zookeeper Pulling 1.3s
• taskmanager Pulling 1.3s
  • e9afc4f90ab0 Pulling fs layer 0.2s
  • 989e6b19a265 Pulling fs layer 0.2s
  • af14b6c2f878 Pulling fs layer 0.2s
  • 68a79816c3e1 Waiting 0.2s
  • 037cc5cb1b83 Waiting 0.2s
  • d3efdb331614 Waiting 0.2s
  • bf82d2b871ad Waiting 0.2s
  • 4ff2e8c5d83f Waiting 0.2s
    [+] Running 0/390 Waiting 0.2s
• postgres Pulling 1.4s
  • 54fec2fa59d0 Waiting 0.2s
  • 30a95add0890 Waiting 0.2s
  • 57bc798d3c84 Waiting 0.2s
  • a41bedb2c5de Waiting 0.2s
  • 589548c3abb4 Waiting 0.2s
  • c4c6b75d5deb Waiting 0.2s
  • 8f8c045a6a99 Waiting 0.2s
  • 69f9dd86b24d Waiting 0.2s
  • 45bbaba740ff Pulling fs layer 0.2s
  • 1761ca7befa0 Waiting 0.2s
  • 57feb34018f4 Pulling fs layer 0.2s
  • bede8373accc Waiting 0.2s
  • 6e4c69fbe63b Waiting 0.2s
  • 8a7949704ab2 Waiting 0.2s
• serverloggenerator Error 1.3s
• frauddetection Error 1.4s
• zookeeper Error 1.4s
• taskmanager Pulling 1.4s
  • e9afc4f90ab0 Pulling fs layer 0.3s
  • 989e6b19a265 Pulling fs layer 0.3s
  • af14b6c2f878 Pulling fs layer 0.3s
  • 68a79816c3e1 Waiting 0.3s
  • 037cc5cb1b83 Waiting 0.3s
  • d3efdb331614 Waiting 0.3s
  • bf82d2b871ad Waiting 0.3s
  • 4ff2e8c5d83f Waiting 0.3s
  • 33cea61c6940 Waiting 0.3s
  • 7a0ed90d1d77 Waiting 0.3s
  • 4be3b2c44e68 Waiting 0.3s
  • 4a884ce80102 Waiting 0.3s
• kafka Pulling 1.4s
  • e7c96db7181b Waiting 0.2s
  • f910a506b6cb Waiting 0.2s
  • b6abafe80f63 Waiting 0.2s
  • 94491bf2b0f8 Waiting 0.2s
  • 6e5322846ab1 Pulling fs layer 0.2s
  • 29422cc83d6d Waiting 0.2s
• jobmanager Pulling 1.4s
  Error response from daemon: pull access denied for sde, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Issue with ververica platform deployed on k8s

Thats a great project!! So far as I know we can create a similar project using Flink SQL. I tried to deploy a flow like this (using ververica platform deployed on k8s): MySQL DB -> kafka -> SQLFlink -> mySQL DB. When doing this, i had problem with deploy on K8s but cannot see any fail log, and i cannot see any records to be written to destination db. So could you please help me more to config this one?
I'll give you more detail about this if you can help! Thanks in advance and have a nice weekend

Thanks and regards!

Make run Not Running Properly on Docker Windows 10

sleep 20
process_begin: CreateProcess(NULL, sleep 20, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [Makefile:11: sleep] Error 2

Enable exactly one delivery

Recommendation from Linkedin

add Flink checkpoints, savepoints, state recovery, and fault tolerance.
We can achieve end-end Exactly once semantics using flink checkpointing and sink transactional semantics combined with Kafka idempotent producer.