thomaspanji / nyc-taxi

Performs data cleaning and manipulation for New York City taxi using PySpark. The resulting data frame then processed to make a visualization using Plotly-Dash.

New York City Green Taxi Data

Introduction

This repository is all about cleaning and analyzing New York City green taxi data. This project contains two main tasks: cleaning raw data using PySpark and building a dashboard for cleaned data using Plotly-Dash.

The data source can be downloaded from this link.

Task 1 : Cleaning

I use Jupyter Notebook and PySpark to clean up the data so I can do a quick analysis of the data and make decisions from it.

For now, I'm only using the 2020 green taxi data. This cleaning process is quite challenging because it gets confusing a lot. In the end, I had to delete a lot of rows. The notebook can be accessed here.

Task 2 : Building a Dashboard

The cleaned data is used to create a dashboard that will display important keys about the business. I use Plotly for plotting purpose combined with Dash to build an interactive dashboard in a web application.

The dashboard is still a work in progress but enough to show some plot there. It took me some effort to understand HTML page styles and settings. Documentation pages are a great way to learn about this amazing thing.

Conclusion

By doing this project, there are some learning I achieved:

1. Running Spark in a notebook
2. Cleaning data by leveraging some PySpark functionalities
3. Knowledge about data format that works efficiently in Spark
4. Plotly and Dash is a good combination for making an interactive web-based dashboard.

This project is not finished yet, I will update all tasks if I find some insight.