Data Processing Pipeline using AWS Kinesis Data Stream (KDS)
- Fetches weather info from data source in JSON format using servier-side-pagination and transforms the data
- Streams the data as batch of 500 records to Kinesis AWS resource.
- Fectched Weather Data (Preceiptation, Snow, Min & Max Temperatures for Maryland for October 2021.
- Listens to Kinesis Data Stream and processes the data by inserting records into Preceiptation and Temparature dynamo DB tables.