fleet-analytics-dashboard / application

A dashboard that supports fleet managers and decision makers to gain insights into their automotive fleets and optimize them

Fleetboard

This project was developed for the course "338025 Data Science Project - Optimizing Fleet Management" of Prof. Dr. Jan Kirenz at the Stuttgart Media University. The dashboard was developed by Lucie, Tim, Larisa, Jakob, Johannes.

Project Goal

This dashboard supports fleet managers and decision makers to gain insights into their automotive fleets and optimize them. The data foundation for this the NREL Fleet DNA project Truck Platooning Data and simulated data by Tim.

It consists of three pages for satisfying the need of different target groups:

• Home view: especially controllers and financial officer can quickly gain insights into the most important key metrics and make high level decisions (e.g. Revenue, Costs, Capacity Overview)
• Downtimes: this page enables fleet managers to easily monitor their fleet and respond to various challenges fast and confidently
• Vehicle overview: if detailed information about certain vehicles or their position in real time on a map is neded, fleet managers can view this information on this page

Wiki page

The wiki has the following sub pages:

1. Home
2. Goals
3. Solution Approach
4. User Research
5. Design Concept
6. Data Foundation
7. Technology stack
8. Data Science
9. Dashboard Explanation
10. Lessons Learned
11. Potential Improvements

Repository Structure

The files in this repository have the following purpose:

• .gitignore: enables to ignore files that should not be pushed like database config file
• apps: all of the dashboard views can be found here
• assets: logo and stylesheet
• csv_data_files: data foundation as CSV files (as a fallback in case of database issues)
• data_preparation: Python scripts for preparing the raw NREL Fleet DNA project Truck Platooning and simulated data
• hidden: needs to be created with a database.config file inside to connect to the project's PostgreSQL database (ask Johannes for database connection information)
• init.py: required to make Python treat the directories as containing packages
• app.yaml: contains the definition of the Python runtime, the entrypoint Gunicorn, and the database connection (need to be specified in the database.config as explained above
• data_preparation_run.py: Python script for running the data preparation
• database_connection.py: Python script for connecting to the database
• main.py: entrypoint of the application
• requirements.txt: consists of all dependencies necessary for running the application

Software Architecture

This is the current software architecture. More information can be found on the(technology stack wiki page:

Lessons Learned

Are more detailed list can be found on the lessons learned wiki page

• Creation of business value from data insights (maintenance prediction)
• Predictions with XGBoost and data transformation with pandas and NumPy
• Deployment of Dash app via Gunicorn HTTP server on Google Cloud App Engine
• Working with different python libraries (e.g. Dash, Plotly. pandas, Numpy)
• Creating a dashboard with user-centric data visualization (different views for different target groups)
• Communication is key (GitHub tasks, video calls, documentation, comments)

Contact

• Lucie
• Tim
• Larisa
• Jakob
• Johannes

License

MIT License - Copyright 2020 ©

application's Issues

Create "Overview" view of design mockup

Based on the wiki page Design Concept

Meeting (17.05.2020 - 2 pm)

• User research
• Research data foundations

Create "Downtimes" view of design mockup

Based on the wiki page Design Concept

Rework routing

• look through other solutions
• fix routing
• merge branch into other branches that need adapted routing

Familiarize with predictions using XGBoost

• supervised machine learning with the Boosted Tree Algorithm)
• Machine learning and statistical methods are explained here: https://scikit-learn.org/stable/
• Decision tree (alternative to linear regression) with random forest extensions and XGboost -> random forest takes average from decision trees (bagging vs boosting ->
• predict categories with classifications (which type of driver advised by professor)
• cluster analysis

Futher literature:

• https://docs.google.com/spreadsheets/d/1bsKFjSC9mMWl5P8PuxvTcYN5jRGNaJviRAX7Bheikg0/edit#gid=0
• algobeans best for starting https://algobeans.com/2016/07/27/decision-trees-tutorial/
• http://faculty.marshall.usc.edu/gareth-james/ISL/
• https://developers.google.com/machine-learning/crash-course
• https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/05.08-Random-Forests.ipynb
• https://xgboost.readthedocs.io/en/latest/tutorials/model.html

Insert tips from professors

Fokus legen

• einfach xgboost nehmen (decision trees), nur den algroithumus anschauen -> damit data science machen
• nächster fokus ist die visualisierung
• NOCHMALS DER FOKUS IST DIE ENTSCHEIDUNGSHILFE (MITTELWERT KÖNNTE REICHEN FÜR DENJENIGEN FALL)
  -> business understanding
  -> welches Problem löse ich damit
  -> welche Entscheidung kornkret damit verbessern
  -> besser dinge ansprechen die man hätte berücksichtigen müssen (gründe angeben)
  -> so kann man lösung weiter ausbauen (scope)
  -> analytics themen reinnehmen
  -> welche library? machine learning und statistische methoden https://scikit-learn.org/stable/

• kategorien vorhersagen klassifikation -> was für fahrertyp -> decision tries, (random forest erweiterung zu decision tries) -> xgboost algorithmen -> gibts fertig, man braucht nur code template
• numberische werte vorhersagen -> regression -> decision tries (alternative lineare regression), (random forest erweiterung zu decision tries) -> random forest nimmt mittelwert von entscheidungsbaum (bagging vs boosting -> mehrere modelle -> jedes modell lernt vom anderen)
• clusteranalyse (muster festeltellen)

Literatur:

• https://docs.google.com/spreadsheets/d/1bsKFjSC9mMWl5P8PuxvTcYN5jRGNaJviRAX7Bheikg0/edit#gid=0
• https://algobeans.com/2016/07/27/decision-trees-tutorial/
• http://faculty.marshall.usc.edu/gareth-james/ISL/

-> algobeans beginnen
-> dann https://developers.google.com/machine-learning/crash-course
-> https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/05.08-Random-Forests.ipynb
-> https://xgboost.readthedocs.io/en/latest/tutorials/model.html

Make MVP deployable

• The solution was this tutorial, due to the use of gunicorn HTTP server: Deploy your Dash application with Google Cloud Platform App Engine
• This short tutorial did work for single pages, but not for a multipage solution: Build a Dash App and Deploy to Google Cloud Platform
  

The proces of deployment was documented in the wiki: https://github.com/Fleet-Analytics-Dashboard/Application/wiki/Development-instructions

Research with batch data foundation

• nrel data -> focus of this project
• master thesis data -> insufficient attributes

Create "Controlling" view of design mockup

Based on the wiki page Design Concept

Discuss updated milestones from professor

• milestones
• presentation
• documentation (particular standard?)

Organize tickets

• Improve milestones
• Set milestones for each task
• Sort tickets

Email to professor (10.05)

• Meeting time and date thursday?
• which solution is best (Dash vs Angular)?

Meeting with Kirenz (28.05.2020)

• show mockup -> looks fine :)
• are views a good idea? -> yes they are!
• show VM on Google Cloud -> seems good
• how should our software architecture look like? -> start with deploying simple Dash app on VM and look from there on for better solutions

Research frontend dashboard solution

Goals

• Identify target groups
• Identify user needs
• Create persona

Search terms

• fleet analytics, fleet dashboard, fleet management, logistics, truck
• competitors (existing dashboards) - Jakob
• decision science, decision psychology, technology acceptance (https://towardsdatascience.com/, (articles of Google's Chief Data Science Cassie Kozyrkov)
• business view, economical value
• interviews (Larisa Bosch fleet management)
• general automotive learnings

Data resources

• quantitative and qualitative surveys
• Google scholar
• online resources

Important questions

• Dash vs Shiny vs Angular (Dash see explanation (here)[https://github.com/Fleet-Analytics-Dashboard/Application/wiki/Technology-stack/]
• can complex data easily be visualized? (only Shiny and Dash)
• difficulty to connect python backend and Angular frontend (most be done mostly manual = quite difficult)

Setup Dash MVP on GitHub

• setup main
• setup sub pages
• create routing
• provide data to all sub pages

Tutorials:

• https://dash.plotly.com/installation
• https://community.plotly.com/t/structuring-a-multi-page-app/13328
• https://community.plotly.com/t/structuring-multi-page-app-template/24097

Enable data cleaning file to work with database

Make Dash Tutorial

@johannesstroebele91 recommends this Dash tutorial: https://dash.plotly.com/installation

Especially the part "make your first Dash app", which is part 2: https://dash.plotly.com/layout

• Larisa
• Lucie
• Jacob

Setup Dash view "Home" (previously "Controlling" view)

• Rename vehicle capacity to vehicle status and overview capacity

• Rename "Controlling" to "Home" view @johannesstroebele91

• Delete old "Home" view @johannesstroebele91

• cost pro vehicle category is wanted in the controlling view

• cost pro category is wanted in the controlling view (wanted by professor)

• would like to only see the list of categories with the highest costs, but be able to click into the category and see the cost for every single truck (wanted by professor) @ts170

• Implement card with numbers: Profit Increase: 76% , Revenue: 58%, Costs -42%, Active: 55%, On Time: 93, In Maintenance 35

• Fix vertical space between cards @johannesstroebele91

• Revise structure of page

  1. Business Goals
  2. Overview of fixed numbers
  3. Costs
  4. Use Vehicle Capacity
  5. Carbon Footprint
  6. Kept Delivery Dates

• there should be an addtional trend line in the bar chart for the business goals (e.g. revenue) (wanted by professor)

• Based on the mockup, search for the relevant data in the "cleaned-data-for-fleet-dna.csv"

• Build the visualizations with Dash based on the identified data

• Exchange better data visualizations for the respective used case based on the following recommendations

  • https://github.com/ft-interactive/chart-doctor/blob/master/visual-vocabulary/Visual-vocabulary.pdf
  • https://echarts.apache.org/examples/en/index.html#chart-type-pie
  • https://datavizproject.com/

Setup organization

• create team
• setup GitHub organization
• setup project
• setup basic wiki
• setup project management process
• arrange group roles

Research Apache Spark - Python

• Google docu https://cloud.google.com/solutions/monte-carlo-methods-with-hadoop-spark
• official python documentation: https://spark.apache.org/docs/latest/quick-start.html
• search for other tutorials about apache spark on YT or udemy

Presentation: create & hold (23.07.20: 14:15)

Information from professor Kirenz:

• show challenges! show what you learned from them
• show a process, not just finished product
• show next steps (what would be important but could not be finished
• 20-25 minutes maximum time for presenting (5 min for feedback)
• each team member must present
• can make a power point presentation but can also show working web dashboard or
• Adesso (Mr. Röhring will be present)
• Show red thread
• Show our focus (technical, business, ..) -> only talk about others briefely
• Show documentation on Github and how to use it (dev instrucations, tasks)
• each group member should presend!

Create "Vehicles Table" view of design mockup

Based on the wiki page Design Concept

Data Cleansing

• setup Python script
• select relevant attributes
• Replace 'clas_id', 'voc_id', 'type_id', 'drive_id' and 'fuel_id' with representing strings from PDF Document
• Check for duplicate content e.g. characteristic acceleration
• improve labels
• fix missing columns (should contain 0s; creating the data frame was none used)

Research possibile prediction use cases

A maximum of 2 predictions as suggested by the professor. We decided on implementing predictive maintenance first, and if there is time left, add other predictions.

• predictive Maintenance (wanted by professor and advisor)
• Different costs (wanted by professor and advisor)
• Categorization?
• Revenue, profit
• dummy data possible
• document how to do that

Researched Dataproc for Apache Spark

Created example "monte carlo" project and deleted it again to save data
https://cloud.google.com/solutions/monte-carlo-methods-with-hadoop-spark

Automate GitHub Kanban

https://medium.com/the-andela-way/how-to-build-a-power-up-for-your-github-project-board-for-project-management-344d5b380a68

Work through professors guideline for deploying app via Google Cloud VM

Instructions here or here with following steps:

• Setup Google Cloud account
• Setup a VM instance
• Make External IP as Static
• Create Firewall Rules
• Install Anaconda
• Connect with Google VM via SSH
• Clone Dash Application from GitHub
• Navigate to directory of the main.py
• Execute the main.py
• If not possible to access via IP, set it manually

Connect application with PostgreSQLdatabase

• setup Python PostgreSQL database psycopgadapter
• change files to data from database into application
• make "data-cleaning.py" work again

Brainstorm software architecture & dev approach for solution

See documentation: https://github.com/Fleet-Analytics-Dashboard/Application/wiki/Technology-stack

• Clarify stages for web app solutions (see documentation)
• choose database for Google Cloud -> PostgreSQL
• choose data science tool -> Pandas and NumPy
• Decide between working with real clusters or dummy environment (different effort needed) -> not until now
• Will not stream data for this project (https://www.tutorialspoint.com/pyspark/index.htm) -> not until now
• Decided on XGBosst for predicting certain outcomes

Fix database connector issues for mac

psycopg2-binary instead of psycopg2 needed in the requirement.txt

Setup Dash view "Vehicles Table"

• callback for filter realtime map (dropdown)
• Implement 2 Tabs to switch between realtime map and vehicle visualization graph
• Table should be always visible underneath
• Based on the mockup, search for the relevant data in the "cleaned-data-for-fleet-dna.csv"
• Build the visualizations with Dash based on the identified data
• Exchange better data visualizations for the respective used case based on the following recommendations
  • https://github.com/ft-interactive/chart-doctor/blob/master/visual-vocabulary/Visual-vocabulary.pdf
  • https://echarts.apache.org/examples/en/index.html#chart-type-pie
  • https://datavizproject.com/

Define goals

• Project goals
• Group member goals

Meeting (30.04.2020)

Meeting (07.05.2020)

• review of research results
• talk group goals
• explore batch data
• talk about technology stack
• define MVP
• plan next tasks

Brainstorm software architecture & development approach for MVP

• select technology stack for MVP solution (local Dash application with panda csv import)
• select technology stack for optimal app solution (deploying app remotely on Google Cloud VM, connecting app with PyCharm)
• document findings in "technology stack" page of the wiki

Setup Dash view "Downtimes"

• enlarge maintenance calender
• Get values for accident probability @ts170
• Implement predictions into calender
• Based on the mockup, search for the relevant data in the "cleaned-data-for-fleet-dna.csv"
• Build the visualizations with Dash based on the identified data
• Exchange better data visualizations for the respective used case based on the following recommendations
  • https://github.com/ft-interactive/chart-doctor/blob/master/visual-vocabulary/Visual-vocabulary.pdf
  • https://echarts.apache.org/examples/en/index.html#chart-type-pie
  • https://datavizproject.com/

PS carbon footprint could be calculated based on speed driven, distance driven, number of stops, ...

Meeting with professor 14.05 (14:15)

Results were written down on the project goals wiki page

• Explain development approach:

• Target group: fleet manager
• Dashboard mit Dash und Python
• Focus on batch processing (later streaming, if enough time is left)
• LAMDA Stack? Database?
• Master thesis data basis first (if enough time city mobility data)

• What is the difference of our project in comparison to the master thesis?

Create & upload documentation (31.07.2020)

Information from professor Kirenz:

• Home
• Goals
• Solution Approach: show work process and progress; CRISP-DM https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining
• Research
• Design concept
• !!! Maintenance Calender!!! @lj025
• Data Science @ts170
• Technology stack @ts170
• Dashboard Explanation @EVERBODY
• Lessons Learned @EVERBODY
• Potential Improvements: next steps (future scope) @EVERBODY

Setup Google Cloud SDK (only for backend)

• Tim
• Johannes

Change attributes names in database

Setup Database on Google VM

Software architecture image can be found here: https://awwapp.com/b/uz6vkdfl3atut/

• create database instance on Google Cloud SQL
• insert raw data from csv file into database table with pgfutter

Documentation:

• Importing data into Cloud SQL
  
• Google Cloud PostgreSQL
• official tutorial Tutorial
• https://codelabs.developers.google.com/codelabs/cloud-vision-app-engine/index.html?index=..%2F..index#0
• https://cloud.google.com/sql/docs/postgres/connect-app-engine-standard
• https://www.fullstackpython.com/postgresql.html#:~:text=PostgreSQL%20is%20the%20default%20database,those%20databases%20are%20reasonable%20choices.

Setup VM on Google Cloud

• get 300$ free credits (@johannesstroebele91 )
• add all members to cloud account

Choose database

PostgreSQL was chosen from these options: PostgreSQL vs MySQL vs SQLite

Create concept for the application

• sitemap based on user roles
• Brainstorm additional features (e.g. alerts)
• Add screenshots of design mockup to the documentation

Research streaming data foundation

Decided against due to lack of time

Create interim presentation

• 01 About the Project
• 02 User Research
• 03 Software Architecture
• 04 Design Concept + Live Demo
• 05 Project Status

Organize wiki and tasks

Write down results into tasks or the wiki of:

• group discussions
• professor's tips
• new features of our app
• new wiki posts

Clarify deadlines & presentation demands (waiting Moodle course)

Meeting with professor (18.06.2020 3:45pm)

• Ask professor about other teams about progress
• Is database solution ok?
• How much data science should we do (descriptive statistics)?

Fix pandas relative path issues for loading csv

Difficult to solve, therefore wait on connecting backend and database (this will automatically solve the probem)
https://daniel.rbind.io/2014/05/16/absolute-and-relative-directories-in-python/