This repository is all about cleaning and analyzing New York City green taxi data. This project contains two main tasks: cleaning raw data using PySpark and building a dashboard for cleaned data using Plotly-Dash.
The data source can be downloaded from this link.
I use Jupyter Notebook and PySpark to clean up the data so I can do a quick analysis of the data and make decisions from it.
For now, I'm only using the 2020 green taxi data. This cleaning process is quite challenging because it gets confusing a lot. In the end, I had to delete a lot of rows. The notebook can be accessed here.
The cleaned data is used to create a dashboard that will display important keys about the business. I use Plotly for plotting purpose combined with Dash to build an interactive dashboard in a web application.
The dashboard is still a work in progress but enough to show some plot there. It took me some effort to understand HTML page styles and settings. Documentation pages are a great way to learn about this amazing thing.
By doing this project, there are some learning I achieved:
- Running Spark in a notebook
- Cleaning data by leveraging some PySpark functionalities
- Knowledge about data format that works efficiently in Spark
- Plotly and Dash is a good combination for making an interactive web-based dashboard.
This project is not finished yet, I will update all tasks if I find some insight.