elsayed91 / 100-days-of-code

Fork this template for the 100 days journal - to keep yourself accountable (multiple languages available)

Hello there! I'm Islam ✨✨

🔧 Technologies & Tools

🌱 About Me

Former manager and civil engineer transitioning to data engineering after discovering a passion for building scalable solutions.

🚀 Featured Projects

• NY Taxi Data & MLOps Pipeline: Automated data & MLOps pipeline leveraging Kubernetes and Apache Airflow. Integrates Spark, Kafka, and DBT with a focus on data quality. Tailors solutions for diverse user needs.
• Xbox Data Scraping & Analysis Pipeline: Automated data-driven project leveraging Python, Airflow, and GKE. Scrapes diverse data sources, providing insights into Xbox hardware and game data.

📦 Packages

• Easy Expectations: A python package that abstracts away the complexity of Great Expectations and allow for easy no-knowledge-required implementation for basic use cases.
• SchemaDiff: A python package that efficiently detects files with inconsistent schemas amidst thousands of files by reading the parquet files metadata.
• Order of The Template: A Python toolkit for parsing and processing YAML templates, capable of resolving Bash syntax environment variables and Jinja templating. It also offers schema validation functionality.