Regression Trees and Model Optimization - Lab
Introduction
In this final lab, we'll see how to apply regression analysis using CART trees for regression while making use of some hyperparameter tuning to improve our model. For a comparison of predictive capabilities and computational cost, we'll work the "Boston Housing" dataset. This will allow us to compare different regression approaches in terms of their accuracy and cost involved.
Objectives
You will be able to:
- Apply predictive regression analysis with CART trees
- Get the data ready for modeling
- Tune the key hyper parameters based a various models developed during training
- Study the impact of tree pruning on the quality of predictions
Boston Housing Dataset - Again !
The dataset is available in the repo as boston.csv
.
- Load the Dataset and print its head and dimensions
# Your code here
Identify Features and Target Data
In this lab, we'll use three features from the Boston housing dataset: 'RM'
, 'LSTAT'
, and 'PTRATIO'
. You'll find a brief description of each predictor below:
Features
'RM'
is the average number of rooms among homes in the neighborhood.'LSTAT'
is the percentage of homeowners in the neighborhood considered "lower class" (working poor).'PTRATIO'
is the ratio of students to teachers in primary and secondary schools in the neighborhood.
Target
-
MEDV
', the median value of the home. -
Create dataframes for features and target as shown above.
-
Inspect the contents for validity
# Your code here
Inspect Correlations
- Use scatter plots to show the correlation between chosen features and target variable
- Comment on each scatter plot
# Your code here
Create Evaluation Metrics
- Create a function
performance(true, predicted)
to calculate and return the r-sqaured score and MSE for two equal sized arrays showing true and predicted values - TEst the function with given data
# Evaluation Metrics
# Import metrics
def performance(y_true, y_predict):
""" Calculates and returns the performance score between
true and predicted values based on the metric chosen. """
# Your code here
pass
# Calculate the performance - TEST
score = performance([3, -0.5, 2, 7, 4.2], [2.5, 0.0, 2.1, 7.8, 5.3])
score
# [0.9228556485355649, 0.4719999999999998]
Supervised Training
- For supervised learning, split the
features
andtarget
datasets into training/test data (80/20). - For reproducibility, use
random_state=42
# Your code here
Grow a Vanilla Regression Tree
- Run a baseline model for later comparison using the datasets created above
- Generate predictions for test dataset and calculate the performance measures using the function created above.
- Use
random_state=45
for tree instance - Record your observations
# Your code here
# (0.4712438851035674, 38.7756862745098) - R2, MSE
Hyperparameter Tuning
- Find the best tree depth for a depth range: 1-30
- Run the regressor repeatedly in a for loop for each depth value.
- Use
random_state=45
for reproducibility - Calculate MSE and r-squared for each run
- Plot both performance measures, for all runs.
- Comment on the output
# Your code here
More Hyperparameter Tuning
-
Repeat the above process for
min_samples_split
parameter -
Use a a range of values from 2-10 for this parameter
-
Use
random_state=45
for reproducibility -
Visualize the output and comment on results as above
# Your code here
Run the "Optimized" Model
- Use the best values for max_depth and min_samples_split found in previous runs and run an optimized model with these values.
- Calculate the performance and comment on the output
# Your code here
Level Up - Optional
- How about bringing in some more features from the original dataset which may be good predictors?
- Also , try tuning more hyperparameters like
max-features
to find the optimal version of the model.
Summary
In this lab, we looked at applying a decision tree based regression analysis on the Boston Housing Dataset. We saw how to train various models to find the optimal values for pruning and limiting the growth of the trees. We also looked at how to extract some rules from visualizing trees , that might be used for decision making later.