esharma3 / project_australian_bushfires_webapp

Detailed analysis of deadly Australian Bushfires, their locations, number of fires, impact caused by the fires and climate & greenhouse gases role in causing the fires, presented in the form of a web based flask app.

AUSTRALIA BUSHFIRE CENTRAL (WebApp) - Group Project

University of Texas at Austin, Data Bootcamp 2019-2020

Group Members - Ekta Sharma | David Altuve | James Durnan | Catherine Gomes | Claudia Ibarra | Austin SpaceK

Project Requirements

Objective:

To analyze and present the history and impact of Australian Bushfires through a web app. Bushfires in Australia are a widespread and regular occurrence that have contributed significantly to moulding the nature of the continent over millions of years. Bushfires have killed approximately 800 people in Australia since 1851, and billions of animals. The purpose of this project is to dig deeper into the roots of current and historical bushfires in Australia and present the findings and analysis results in the form of a web based Flask app with routes for - Australian bushfire locations, global bushfire locations, current and historical fire counts, impact caused by bushfires and climate’s role in causing the bushfires, home and about page.

Additional Highlights:

• Access to the project data using API requests
• Fun 404 page
• Feedback section on the Home Page that lets user post comments

Technologies Used:

• APP – Flask, Flask-SQLAlchemy, Python
• Database – MySQL on GCP. Script created using SQL
• Templates – HTML, CSS, Bootstrap
• Interactive Features – JavaScript, Plotly
• Visualizations – Plotly map, bar graph, pie chart, line graph, scatter plot
• Web Scraping – Requests, Beautiful Soup
• Deployment - GCP (Google Cloud Platform)

Data Sources:

• Australia and Global Fire Locations – NASA, Fire Information for Resource Management System (FIRMS)
• Protected Species Impact - Australian Government, Department of Agriculture, Water and the Environment
• Socio - Economic Impact – Wikipedia Economy of Australia
  • Bushfire Season – Wikipedia 2019–20 Australian bushfire season
  • Major Bushfires – Wikipedia, List of major bushfires in Australia
  • The World Bank Data, Country Australia
• Climate Data – Australian Government Bureau of Meteorology
• Greenhouse Gases – CSIRO
• Fire Counts – GFED (Global Fire Emissions Database)

Project Data Routes:

• https://australia-bushfire-central-ak7h3hm6ya-uc.a.run.app/annual_total_fire_counts?state=&year=
• https://australia-bushfire-central-ak7h3hm6ya-uc.a.run.app/climate_data?category=
• https://australia-bushfire-central-ak7h3hm6ya-uc.a.run.app/impact-data
• https://australia-bushfire-central-ak7h3hm6ya-uc.a.run.app/econ-impact
• https://australia-bushfire-central-ak7h3hm6ya-uc.a.run.app/aus_fire_map/

Project API Documentation:

https://documenter.getpostman.com/view/10893300/SzYXVdtw

© 2020 Copyright: University of Texas, Data Bootcamp

project_australian_bushfires_webapp's Issues

Create route for fire counts template in app.py

Update the load_my_sql.py global clilmate query

Create Routes for Global Fires Data

This issue represents my progress on building routes for the Global Fires Data.

Create a page displaying the map of australia with pins

Create a page using the default template to display the map of australia with pins corresponding to the australia fire archives

Kangaroo Island/National Parks Data

Find all bushfire data (fires, animals, firefighters, anything that would make sense to add to our KI template) related to Kangaroo Island or other National Parks. The data could be in the form of csv files or can be scraped.

Create data and page route for climate fails template in app.py.

Fix SQL script to include desired dtypes prior to importing table to SQL

Fire Counts Data

Find a data source for Australian Fire counts and add the data files to the data folder in the repo.

create route to contain aus_fire_history data and return as json

this route will be used to source the australia map displaying fire history

Create custom 404 page

Climate Data

Find Australia's climate data that can be correlated to the Australian bushfires and can be shown on graph as a reason contributing towards the bushfires.

Add climate and air pollutant data tables to the database

Add climate and air pollutant data tables to the database.

Add images to about page

Create ReadMe File.

fix typo in load_to_mysql.py

correct route name for fire maps

Finish Global Fires App and JS code

This is the issue for finishing these portions of the project for the global fires data.

Create Climate Fails Page/Template

Add Fire Count Tables to the database.

Add fire count table(s) for the 3 states (NSW, Queensland, Victoria) to the database.

Deploy the app on GCP.

Project Proposal Documentation

Create a one page project proposal document and submit before official project start date. The document will clearly describe the scope of the project.

Fix class name to match table name

Create bushfires_db and add tables for fire count.

Create Virtual Environment

Create the pip and pip lock file with the required packages for the project and upload it to Github.

Writing several large data files to SQL

Working with the data has required careful handling to reduce memory usage. My remaining project work will be designed around this theme.

This issue will track my progress through chunking 2018 + 2017 data and writing it to MySQL.

Create .env file and add database connection string and passwords to it. Install python-dotenv and connect to the database from the app using environment variable.

Improve climate-fails and fire-count page's design/layout/content (using javascript).

Create bushfires_db and upload fire-count and climate tables on GCP

Manipulate/Clean Large Data for Global Fire Map

I am in the process of finding a way to to load large chunk data into a pandas data frame. This requires chunking the data set into several pieces. In order to create the data frame, memory usage needs to be calculated.

Wishlist Item - Create API routes for climate and fire-count data.

Upload all the tables for the entire app to GCP database. Test it.

Bushfire Impacts Data

Find all the data that can be represented as 'IMPACTS of BUSHFIRE' in Australia. The impacts could be on human, wildlife, economic, psychological etc. The data could be in the form of csv, API or can be scraped.

Reload Data Table in MySQL with additional months

Before continuing on with the routes, templates, and .js file, I will go back and iterate through 4 cleaned csv files. The iterated data in the to_my_sql.py file will contain updated date ranges.

Code for scraping, cleaning and combining climate and air pollutant data

Write the code for scraping, cleaning and combining climate and air pollutant data. Create csv files with this data that will be uploaded to MySQL database.

Global Bushfires Locations Data

Find global bushfire locations data and add it to the data folder. The data must be from Jan 2015 till Feb 2020 and must have latitudes and longitudes for the fire locations.

Create template for fire counts page

Australian Bushfire Locations data

Find Australian bushfire locations data and add it to the data folder. The data must be from Jan 2015 till Feb 2020 and must have latitudes and longitudes for the fire locations.

Create Master Template

Create the master template that will be used by everyone when building their individual templates (web page for each route). This master template should have the navbar with all the route names and should be reviewed and approved by all group member.