learn-co-curriculum / ds-pandas-summative-lab

Pandas Summative Lab

Problem Statement

In this lab, we'll make use of everything we've learned about pandas, data cleaning, and Exploratory Data Analysis.

The Dataset

In this lab, we'll work with the comprehensive Super Heroes Dataset, which can be found on Kaggle!

Objectives

• Use all available pandas knowledge to clean the dataset and deal with null values
• Use Queries and aggregations to group the data into interesting subsets as needed
• Use descriptive statistics and data visualization to find answers to questions we may have about the data.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
pd.options.display.max_rows = 1000
%matplotlib inline

For this lab, our dataset is split among two different sources--heroes_information.csv and super_hero_powers.csv.

Use pandas to read in each file and store them in DataFrames in the appropriate variables below. Then, display the head of each to ensure that everything loaded correctly.

heroes_df = None
powers_df = None

It looks as if the heroes information dataset contained an index column. We did not specify that this dataset contained an index column, because we hadn't seen it yet. Pandas does not know how to tell apart an index column from any other data, so it stored it with the column name Unnamed: 0.

Our DataFrame provided row indices by default, so this column is not needed. Drop it from the DataFrame in place in the cell below, and then display the head of heroes_df to ensure that it worked properly.

Familiarize Yourself With the Dataset

The first step in our Exploratory Data Analysis will be to get familiar with the data. This step includes:

• Understanding the dimensionality of your dataset
• Investigating what type of data it contains, and the data types used to store it
• Discovering how missing values are encoded, and how many there are
• Getting a feel for what information it does and doesnt contain

In the cell below, get the descriptive statistics of each DataFrame.

Dealing with Null Values

Starting in the cell below, detect and deal with any null values in either data frame. Then, explain your methodology for detecting and dealing with outliers in the markdown section below. Be sure to explain your strategy for dealing with null values in numeric columns, as well as your strategy for dealing with null values in non-numeric columns.

Note that if you need to add more cells to write code in, you can do this by:

1. Highlighting a cell and then pressing ESC to enter command mode.
2. Press A to add a cell above the highlighted cell, or B to add a cell below the highlighted cell.

Describe your strategy below this line:

---

Joining, Grouping, and Aggregating

In the cell below, join the two DataFrames. Think about which sort of join you should use, as well as which columns you should join on. Rename columns and manipulate as needed.

HINT: If the join throws an error message, consider setting the column you want to join on as the index for each DataFrame.

In the cell below, create a scatterplot of the height and weight of each hero, with weight as the y-axis. Color each point in the scatterplot according to the gender of the superhero.

Next, slice the DataFrame as needed and visualize the distribution of heights and weights by gender. You should have 4 total plots.

male_heroes_df = None
female_heroes_df = None

def show_distplot(dataframe, gender, column_name):
    pass

# Male Height

# Male Weight

# Female Height

# Female Weight

Discuss your findings from the plots above, with respect to the distibution of height and weight by gender. Your explanation should include discussion of any relevant summary statistics, including mean, median, mode, and the overall shape of each distribution.

Wite your answer below this line:

---

Sample Question: Most Common Powers

The rest of this notebook will be left to you to investigate the dataset by formulating your own questions, and then seeking answers using pandas and numpy. Every answer should include some sort of visualization, when appropriate. Before moving on to formulating your own questions, use the dataset to answer the following questions about superhero powers:

• What are the 5 most common powers overall?
• What are the 5 most common powers in the Marvel Universe?
• What are the 5 most common powers in the DC Universe?

def top_5_powers(dataframe):
    pass
    
overall_top_5 = None
marvel_df = None
dc_df = None
print(overall_top_5)

marvel_top_5 = None
print(marvel_top_5)

dc_top_5 = None
print(dc_top_5)

def top_5_bar_chart(top_5_list, publisher=None):
    pass    

Analyze the results you found above to answer the following question:

How do the top 5 powers in the Marvel and DC universes compare? Are they similar, or are there significant differences? How do they compare to the overall trends in the entire Superheroes dataset?

Wite your answer below this line:

---

Your Own Investigation

For the remainder of this lab, you'll be focusing on coming up with and answering your own question, just like we did above. Your question should not be overly simple, and should require both descriptive statistics and data visualization to answer. In case you're unsure of what questions to ask, some sample questions have been provided below.

Pick one of the following questions to investigate and answer, or come up with one of your own!

• Which powers have the highest chance of co-occuring in a hero (e.g. super strength and flight), and does this differ by gender?
• Is there a relationship between a hero's height and weight and their powerset?
• What is the distribution of skin colors amongst alien heroes?

Explain your question below this line:

---

Some sample cells have been provided to give you room to work. If you need to create more cells, you can do this easily by:

1. Highlighting a cell and then pressing esc to enter command mode.
2. Pressing b to add a cell below the currently highlighted cell, or a to add one above it.

Be sure to include thoughtful, well-labeled visualizations to back up your analysis!

Conclusion

In this lab, we demonstrated our mastery of:

• Use all available pandas knowledge to clean the dataset and deal with null values
• Use Queries and aggregations to group the data into interesting subsets as needed
• Use descriptive statistics and data visualization to find answers to questions we may have about the data.