satyam245 Goto Github PK

100-days-of-machine-learning

airflow-logistics-data-pipeline

Streamline logistics data orchestration with Apache Airflow on Google Cloud Platform. Automate ingestion, transformation, and storage of CSV files in Google Cloud Storage (GCS) into Hive tables on Google Cloud Dataproc. Utilizes dynamic partitioning for scalability and efficiency.

aws-airline-ingestion-pipeline

Efficiently ingest daily airline data into AWS using a seamless end-to-end pipeline, integrating S3 uploads, Glue schema discovery, Redshift data transformation, and SNS notifications.

aws-cloud-project

‘Save To the Cloud’ is a full stack web application that mainly deals with storing and saving files by leveraging cloud infrastructure.

data-engineering-essentials

data-engineering-projects

Personal Data Engineering Projects

data-engineering-with-aws

Data Engineering with AWS, Published by Packt

data-science-ipython-notebooks

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

dataengineering-youtube-analysis-project

Data Engineering YouTube Analysis Project by Darshil Parmar

dataengineeringcapstoneproject

😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS

ecommerce-integration-pipeline

🌐 Seamless E-commerce Data Integration Pipeline using Python, GCP Pub/Sub, DataStax Cassandra, and Pandas. This repository includes scripts to load, publish, and consume data, along with instructions for setting up the environment. Simplify your data integration process for e-commerce orders with this efficient and scalable solution.

etl-pipeline-bank-transaction

Transform daily bank transactions effortlessly with this AWS ETL pipeline. Ingest CSVs to S3, trigger Glue jobs with Lambda, store securely in Parquet, and analyze seamlessly using Athena

free-data-engineering-course-for-beginners

Learn the entire ETL process based on Spotify API data

it-cert-automation-practice

Google IT Automation with Python Professional Certificate - Practice files

kafka-mongo-logistics-integration

A Python application integrating Kafka and MongoDB for efficient logistics data processing, with Avro serialization, Docker scaling, and an API for seamless interaction.

order-tracking-incremental-load-project

The Order Tracking Incremental Load Project automates the integration of order tracking data using Apache Spark in Databricks. Leveraging Google Cloud Storage (GCS) for input, it features efficient stage processing, upserts to a target Delta table, and automated execution.

realtime-orders-data-processing

"Real-Time Data Processing with GCP Pub/Sub and DataStax Cassandra" is a project demonstrating the integration of Google Cloud Platform's Pub/Sub and DataStax Cassandra for efficient real-time data processing. It handles orders and payments data streams, ingesting them via Pub/Sub and storing them in Cassandra tables.

stock-market-kafka-pipeline

Stock Market Kafka Project: A robust data pipeline leveraging Python, Confluent Kafka, AWS S3, IAM, Glue, and Athena to efficiently process and analyze stock market data. Streamline your workflow from CSV ingestion to insights with this comprehensive solution.

udacity-data-engineering-projects

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.