rubenandrebarreiro / apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree

⌚ 🧳 ✈️ A project based in High Performance Computing. This project was built using Apache Spark API, Java and Gradle. The scenario of the project was a Cluster-based implementation of the Prim's Algorithm in a Graph representation of a network of routes between several airports and the average departure delays of that routes. The goal of this project is to find the bottleneck airport, i.e., the airport with greater aggregated departure delays, then apply a reduction factor and recompute the Minimum Spanning Tree (M.S.T. - Prim's Algorithm).

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm)

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Banner #1

---

Current Status

Current Progress of the Project

Final Approval Grade of the Project

Description

A 1st year's lab work (project) of the MSc. degree of Computer Science and Engineering made in FCT NOVA (Faculty of Sciences and Technology of New University of Lisbon), in the subject of High Performance Computing. This project was built using Apache Spark API, Java and Gradle. The scenario of the project was a Cluster-based implementation of the Minimum Spanning Tree (Prim's algorithm), with the support of Apache Spark's API, using its offered cluster-based parallel optimisations and operations.

Click here to see the Project's Description!

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Project's Description #1

---

Screenshots

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Screenshot #1

---

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Screenshot #2

---

Getting Started

Prerequisites

To install and run this application, you will need:

The Git feature and/or a third-party Git Client based GUI, like:

• GitHub Desktop, GitKraken, SourceTree or TortoiseGit.

Installation

To install this application, you will only need to download or clone this repository and run the application locally:

You can do it downloading the .zip file in download section of this repository.

Or instead, by cloning this repository by a Git Client based GUI, using HTTPS or SSH, by one of the following link:

• HTTPS:

https://github.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

• SSH:

[email protected]:rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

Or even, by running one of the following commands in a Git Bash Console:

• HTTPS:

git clone https://github.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

• SSH:

git clone [email protected]:rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

Built with

• Apache Spark API
• Java
• Gradle
• Eclipse IDE
• JetBrains IntelliJ IDEA

Notes

• There's some useful parameters which you can try in as JVM (Java Virtual Machine) parameters, for a better Memory and Java Heap Space management, in the case of be running the application locally, instead of running the application in a cluster system:

Option #1

-Xms8g
-Xmx16g
-XX:MinHeapFreeRatio=50
-XX:MaxHeapFreeRatio=50
-XX:NewRatio=3
-XX:NewSize=2048m
-XX:MaxNewSize=2048m
-XX:+AggressiveHeap

Option #2

-Xms12g
-Xmx16g
-XX:MinHeapFreeRatio=50
-XX:MaxHeapFreeRatio=50
-XX:NewRatio=3
-XX:NewSize=2g
-XX:MaxNewSize=2g
-XX:+AggressiveHeap

Contributors

Rúben André Barreiro

Contacts

Rúben André Barreiro

E-mails

• [email protected]
• [email protected]
• [email protected]
• [email protected]

Portfolios/Blogs and Git Hosting/Repository Services

Rúben André Barreiro

GitHub's Portfolio/Personal Blog

• https://rubenandrebarreiro.github.io/

Hosting/Repository Services

• https://github.com/rubenandrebarreiro/
• https://gitlab.com/rubenandrebarreiro/
• https://bitbucket.org/rubenandrebarreiro/
• https://dev.azure.com/rubenandrebarreiro/

Classifications/Final Approval Grades

Approval Grade of Lab Work

• 18 of 20