In this lab, we shall apply regression analysis using simple matrix manipulations to fit a model to given data, and then predict new values for previously unseen data. We shall follow the approach highlighted in previous lesson where we used numpy to build the appropriate matrices and vectors and solve for the
In order to make this experiment interesting. We shall use NumPy at every single stage of this experiment i.e. loading data, creating matrices, performing test train split, model fitting and evaluations.
You will be able to:
- Use linear algebra to apply simple regression modeling in Python and NumPy only
- Apply train/test split using permutations in NumPy
- Use matrix algebra with inverses and dot products to calculate the beta
- Make predictions from the fitted model using previously unseen input features
- Evaluate the fitted model by calculating the error between real and predicted values
First let's import necessary libraries
import csv # for reading csv file
import numpy as np
The dataset we will use for this experiment is "Sales Prices in the City of Windsor, Canada", something very similar to the Boston Housing dataset. This dataset contains a number of input (independent) variables, including area, number of bedrooms/bathrooms, facilities(AC/garage) etc. and an output (dependent) variable, price. We shall formulate a linear algebra problem to find linear mappings from input to out features using the equation provided in the previous lesson.
This will allow us to find a relationship between house features and house price for the given data, allowing us to find unknown prices for houses, given the input features.
A description of dataset and included features is available at THIS LINK
In your repo, the dataset is available as windsor_housing.csv
containing following variables:
there are 11 input features (first 11 columns):
lotsize bedrooms bathrms stories driveway recroom fullbase gashw airco garagepl prefarea
and 1 output feature i.e. price (12th column).
The focus of this lab is not really answering a preset analytical question, but to learn how we can perform a regression experiment, similar to one we performed in statsmodels, using mathematical manipulations. So we we wont be using any Pandas or statsmodels goodness here. The key objectives here are to a) understand regression with matrix algebra, and b) Mastery in NumPy scientific computation.
Let's give you a head start by importing the dataset. We shall perform following steps to get the data ready for analysis:
-
Initialize an empty list
data
for loading data -
Read the csv file containing complete (raw)
windsor_housing.csv
. Usecsv.reader()
for loading data.. Store this indata
one row at a time. -
Drop the first row of csv file as it contains the names of variables (header) which won't be used during analysis (keeping this will cause errors as it contains text values).
-
Append a column of all 1s to the data (bias) as the first column
-
Convert
data
to a numpy array and inspect first few rows
NOTE: read.csv()
would read the csv as a text file, so we must convert the contents to float at some stage.
# Your Code here
# First 5 rows of raw data
# array([[1.00e+00, 5.85e+03, 3.00e+00, 1.00e+00, 2.00e+00, 1.00e+00,
# 0.00e+00, 1.00e+00, 0.00e+00, 0.00e+00, 1.00e+00, 0.00e+00,
# 4.20e+04],
# [1.00e+00, 4.00e+03, 2.00e+00, 1.00e+00, 1.00e+00, 1.00e+00,
# 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
# 3.85e+04],
# [1.00e+00, 3.06e+03, 3.00e+00, 1.00e+00, 1.00e+00, 1.00e+00,
# 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
# 4.95e+04],
# [1.00e+00, 6.65e+03, 3.00e+00, 1.00e+00, 2.00e+00, 1.00e+00,
# 1.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
# 6.05e+04],
# [1.00e+00, 6.36e+03, 2.00e+00, 1.00e+00, 1.00e+00, 1.00e+00,
# 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
# 6.10e+04]])
Explore NumPy's official documentation to manually split a dataset using numpy.random.shuffle()
, numpy.random.permutations()
or using simple resampling method.
- Perform a RANDOM 80/20 split on data using a method of your choice , in NumPy using one of the methods above.
- Create x_test, y_test, x_train and y_train arrays from the split data.
- Inspect the contents to see if the split performed as expected.
# Your code here
# Split results
# Raw data Shape: (546, 13)
# Train/Test Split: (437, 13) (109, 13)
# x_train, y_train, x_test, y_test: (437, 12) (437,) (109, 12) (109,)
With our X and y in place, We can now compute our beta values with x_train and y_train as:
- Using numpy operations (transpose, inverse) that we saw earlier, compute the above equation in steps.
- Print your beta values
# Your code here
# Calculated beta values
# [-3.07118956e+03 2.13543921e+00 4.04283395e+03 1.33559881e+04
# 5.75279185e+03 7.82810082e+03 3.73584043e+03 6.51098935e+03
# 1.28802060e+04 1.09853850e+04 6.14947126e+03 1.05813305e+04]
[7.13503312e+03 3.14678249e+00 3.28225971e+02 1.28839251e+04
6.68332755e+03 3.44682885e+03 3.19440776e+03 5.07824499e+03
1.32822228e+04 1.10098716e+04 2.88253206e+03 9.16916600e+03]
Great , we now have a set of coefficients that describe the linear mappings between X and y. We can now use the calculated beta values with the test datasets that we left out to calculate y predictions. For this we need to perform the following tasks:
Now we shall all features in each row in turn and multiply it with the beta computed above. The result will give a prediction for each row which we can append to a new array of predictions.
- Create new empty list (y_pred) for saving predictions.
- For each row of x_test, take the dot product of the row with beta to calculate the prediction for that row.
- Append the predictions to y_pred.
- Print the new set of predictions.
# Your code here
This is exciting, so now our model can use the beta value to predict the price of houses given the input features. Let's plot these predictions against the actual values in y_test to see how much our model deviates.
# Plot predicted and actual values as line plots
This doesn't look so bad, does it ? Our model, although isn't perfect at this stage, is making a good attempt to predict house prices although a few prediction seem a bit out. There could a number of reasons for this. Let's try to dig a bit deeper to check model's predictive abilities by comparing these prediction with actual values of y_test individually. That will help us calculate the RMSE value (Root Mean Squared Error) for our model.
Here is the formula for this again.
- Initialize an empty array
err
. - for each row in y_test and y_pred, take the squared difference and append error for each row in err array.
- Calculate RMSE from
err
using the formula shown above.
# Calculate RMSE
# Due to random split, your answers may vary
# RMSE = 16401.913562758735
The above error is clearly in terms of the dependant variable i.e. the final house price. We can also use a normlized mean squared error in case of multiple regression which can be calculated from RMSE using following formula:
- Calculate normalized Root Mean Squared Error
# Calculate NRMSE
# Due to random split, your answers may vary
# 0.09940553674399233
0.09940553674399233
SO there it is. A complete multiple regression analysis using nothing but numpy. Having good programming skills in numpy would allow to dig deeper into analytical algorithms in machine learning and deep learning. Using matrix multiplication techniques we saw here, we can easily build a whole neural network from scratch.
- Calculated the R_squared and adjusted R_squared for above experiment.
- Plot the residuals (similar to statsmodels) and comment on the variance and heteroscedascticity.
- Run the experiment in statsmodels and compare the performance of both approaches in terms of computational cost.
So there we have it. A predictive model for predicting house prices in a given dataset. Remember this is a very naive implementation of regression modeling. The purpose here was to get an introduction to the applications of linear algebra into machine learning and predictive analysis. We still have a number of shortcomings in our modeling approach and we can further apply a number of data modeling techniques to improve this model.