target-ready-2023 / datastreaming_team1

Target Ready - Spark/Scala - Team 1 - Kafka Data Streaming

datastreaming_team1

Target Ready - Spark/Scala - Team 1 - Kafka Data Streaming

Data Streaming - Kafka

Use case:

Produce a kafka topic,post messages and consume data.

Steps:

1. Create a dataset of 1 million records and store it in your local machine.
2. Read the dataset and store data in any database (Postgress/Mongo).
3. Create a Kafka topic.
4. Read the dataset using spark and post these 1 million rows as messages into the Kafka topic.
5. Read the messages from the topic and store it any database or text format.

datastreaming_team1's Issues

Team 2 - Load valid records to dataset

• Create 3 datasets as mentioned in the architecture diagram
• Remove duplicates
• Load valid records into the datasets

Team 1 - Create Kafka Topic

Create a Kafka cluster locally
Create a Kafka topic

Team 4 - Produce message & Post in Kafka

Extract data from source dataset
Produce messages and publish it into Kafka topic

Team 1 - Produce message & Post in Kafka

Extract data from source dataset.
Produce messages and publish it into Kafka topic.

Team 4 - Tech Setup - Local IDE

Installation of required applications
Testing

Team 1 - Understand - Kafka & Use case

Read about Kafka.
Learn and understand about the use case in detail

Team 4 - Load valid records to datasets

• Create 3 datasets as mentioned in the architecture diagram
• Remove duplicates
• Load valid records into the datasets

Team 4 - Create Kafka Topic

Create a Kafka cluster locally
Create a Kafka topic

Team 4 - Source dataset creation

Create a source dataset - Product and load data

Team 2 - Consume messages and validate business rules.

1. Consume messages from Kafka topic.

2. Validate business rules

   • Check for selling channel - Move messages with different selling channel other than ONLINE,STORE,CROSSOVER into an error table.
   • Check price is not null or 0.If so, move it to error table.
   • Product id should be only numeric and 8 digit.

Team 3 - Create Kafka Topic

Create a Kafka cluster locally.
Create a Kafka topic.

Team 3 - Source dataset creation

Create a source dataset - Product and load data

Team 2 - Create Kafka Topic

Create a Kafka cluster locally
Create a Kafka topic

Team 1 - Load valid records to dataset

• Create 3 datasets as mentioned in the architecture diagram
• Remove duplicates
• Load valid records into the datasets

Team 3 - Produce message & Post in Kafka

Extract data from source dataset
Produce messages and publish it into Kafka topic

Team 4 - Consume message and validate business rules

1. Consume messages from Kafka topic.

2. Validate business rules

   • Check for selling channel - Move messages with different selling channel other than ONLINE,STORE,CROSSOVER into an error table.
   • Check price is not null or 0.If so, move it to error table.
   • Product id should be only numeric and 8 digit.

Team 2 - Source dataset creation

Create a source dataset - Product and load data

Team 3 - Load valid records to datasets

• Create 3 datasets as mentioned in the architecture diagram
• Remove duplicates
• Load valid records into the datasets

Team 4 - Tech Setup - Local IDE

Installation of required applications
Testing.

Team 2 - Tech Setup - Local IDE

Installation of required applications
Testing.

Team 3 - Tech Setup - Local IDE

Installation of required applications
Testing.

Team 2 - Produce message & Post in Kafka

Extract data from source dataset
Produce messages and publish it into Kafka topic

Team 2 - Understand - Kafka & Use case

Read about Kafka.
Learn and understand about the use case in detail

Team 4 - Understand - Kafka & Use case

Read about Kafka.
Learn and understand about the use case in detail

Team 1 - Tech Setup - Local IDE

Installation of required applications
Testing.

Team 1 - Source dataset creation

Create a source dataset - Product and load data.

Team 1 - Consume messages and validate business rules

1. Consume messages from Kafka topic.

2. Validate business rules

   • Check for selling channel - Move messages with different selling channel other than ONLINE,STORE,CROSSOVER into an error table.
   • Check price is not null or 0.If so, move it to error table.
   • Product id should be only numeric and 8 digit.

Team 3 - Understand - Kafka & Use case

Read about Kafka.
Learn and understand about the use case in detail

Team 3 - Consume messages and validate business rules

1. Consume messages from Kafka topic.

2. Validate business rules

   • Check for selling channel - Move messages with different selling channel other than ONLINE,STORE,CROSSOVER into an error table.
   • Check price is not null or 0.If so, move it to error table.
   • Product id should be only numeric and 8 digit.