elsayed91 / schemadiff

Efficiently groups Parquet files based on their schema, allowing for the detection of schema inconsistencies between files on different filesystems. Achieves high efficiency by reading metadata without loading entire files.

Hello there! I'm Islam ✨✨

🔧 Technologies & Tools

🌱 About Me

Former manager and civil engineer transitioning to data engineering after discovering a passion for building scalable solutions.

🚀 Featured Projects

• NY Taxi Data & MLOps Pipeline: Automated data & MLOps pipeline leveraging Kubernetes and Apache Airflow. Integrates Spark, Kafka, and DBT with a focus on data quality. Tailors solutions for diverse user needs.
• Xbox Data Scraping & Analysis Pipeline: Automated data-driven project leveraging Python, Airflow, and GKE. Scrapes diverse data sources, providing insights into Xbox hardware and game data.

📦 Packages

• Easy Expectations: A python package that abstracts away the complexity of Great Expectations and allow for easy no-knowledge-required implementation for basic use cases.
• SchemaDiff: A python package that efficiently detects files with inconsistent schemas amidst thousands of files by reading the parquet files metadata.
• Order of The Template: A Python toolkit for parsing and processing YAML templates, capable of resolving Bash syntax environment variables and Jinja templating. It also offers schema validation functionality.