andreihushcha / dsc-categorical-variables-regression-lab

Dealing with Categorical Variables - Lab

Introduction

In this lab, you'll explore the Ames Housing dataset and identify numeric and categorical variables. Then you'll transform some categorical data and use it in a multiple regression model.

Objectives

You will be able to:

• Determine whether variables are categorical or numeric
• Use one-hot encoding to create dummy variables

Step 1: Load the Ames Housing Dataset

Import pandas, and use it to load the file ames.csv into a dataframe called ames. If you pass in the argument index_col=0 this will set the "Id" feature as the index.

# Your code here - load the dataset

Visually inspect ames (it's ok if you can't see all of the columns).

# Your code here

Go ahead and drop all columns with missing data, to simplify the problem. Remember that you can use the dropna method (documentation here).

# Your code here - drop columns with missing data

Step 2: Identify Numeric and Categorical Variables

The file data_description.txt, located in this repository, has a full description of all variables.

Using this file as well as pandas techniques, identify the following predictors:

1. A continuous numeric predictor
2. A discrete numeric predictor
3. A string categorical predictor
4. A discrete categorical predictor

(Note that SalePrice is the target variable and should not be selected as a predictor.)

For each of these predictors, visualize the relationship between the predictor and SalePrice using an appropriate plot.

Finding these will take some digging -- don't be discouraged if they're not immediately obvious. The Ames Housing dataset is a lot more complex than the Auto MPG dataset. There is also no single right answer here.

Continuous Numeric Predictor

# Your code here - continuous numeric predictor

Discrete Numeric Predictor

# Your code here - discrete numeric predictor

String Categorical Predictor

# Your code here - string categorical predictor

Discrete Categorical Predictor

# Your code here - discrete categorical predictor

Step 3: Build a Multiple Regression Model with Your Chosen Predictors

Choose the best-looking 3 out of 4 predictors to include in your model.

Make sure that you one-hot encode your categorical predictor(s) (regardless of whether the current data type is a string or number) first.

# Your code here - prepare X and y, including one-hot encoding

# Your answer here - which category or categories were dropped?

# Your code here - build a regression model and display results

Step 4: Create Partial Regression Plots for Features

For each feature of the regression above (including the dummy features), plot the partial regression.

# Your code here - create partial regression plots

Step 5: Calculate an Error-Based Metric

In addition to the adjusted R-Squared that we can see in the model summary, calculate either MAE or RMSE for this model.

# Your code here - calculate an error-based metric

Step 6: Summarize Findings

Between the model results, partial regression plots, and error-based metric, what does this model tell you? What would your next steps be to improve the model?

# Your answer here

Level Up (Optional)

Try transforming X using scikit-learn and fitting a scikit-learn linear regression as well. If there are any differences in the result, investigate them.

# Your code here

Summary

In this lab, you practiced your knowledge of categorical variables on the Ames Housing dataset! Specifically, you practiced distinguishing numeric and categorical data. You then created dummy variables using one hot encoding in order to build a multiple regression model.