jwarmenhoven / coursera-machine-learning

Coursera Machine Learning - Python code

Coursera Machine Learning

This repository contains python implementations of certain exercises from the course by Andrew Ng.

For a number of assignments in the course you are instructed to create complete, stand-alone Octave/MATLAB implementations of certain algorithms (Linear and Logistic Regression for example). The rest of the assignments depend on additional code provided by the course authors. For most of the code in this repository I have instead used existing Python implementations like Scikit-learn.

Exercise 1 - Linear Regression
Exercise 2 - Logistic Regression
Exercise 3 - Multi-class Classification and Neural Networks
Exercise 4 - Neural Networks Learning
Exercise 5 - Regularized Linear Regression and Bias v.s. Variance
Exercise 6 - Support Vector Machines
Exercise 7 - K-means Clustering and Principal Component Analysis
Exercise 8 - Anomaly Detection and Recommender Systems

References:

https://www.coursera.org/learn/machine-learning/home/welcome

coursera-machine-learning's Issues

No `theta_0` parameter used in Exercise 2 Regularized Logistic Regression

Hi.
I am right now doing Andrew Ng. class (Machine Learning) and am using your answer as a guidance. I've realized something in your Exercise 2 (the regularized exercise one).

You've extracted your data from the ex2data2.txt like this:

data2 = loaddata('data/ex2data2.txt', ',')
y = np.c_[data2[:,2]]
X = data2[:,0:2]

And I believe you didn't inserted the theta_0 parameter to the X variable's first column (supposed to be the value of 1). Aren't it supposed to be so? In the non-regularized logistic regression exercise, you did inserted the theta_0 parameter as like this:

X = np.c_[np.ones((data.shape[0],1)), data[:,0:2]]

Am I missing something here?

Optimization parameters

• Method
• Gradient function

Regarding regularized NN equation in exercise 4

I am referring to your exercise 4 code right now to complete mine. I think you've made a mistake on regularizing the NN backpropagation gradient (if I am wrong, pardon me). This is the equation:

And this is your code:

delta1 = d2.dot(a1) # 25x5000 * 5000x401 = 25x401
delta2 = d3.T.dot(a2) # 10x5000 *5000x26 = 10x26
    
theta1_ = np.c_[np.ones((theta1.shape[0],1)),theta1[:,1:]]
theta2_ = np.c_[np.ones((theta2.shape[0],1)),theta2[:,1:]]
    
theta1_grad = delta1/m + (theta1_*reg)/m
theta2_grad = delta2/m + (theta2_*reg)/m

Shouldn't it be

theta1_ = np.c_[np.zeros((theta1.shape[0],1)),theta1[:,1:]]
theta2_ = np.c_[np.zeros((theta2.shape[0],1)),theta2[:,1:]]

since we do not want to add anything to theta1_grad's and theta2_grad's first column (the bias)?

Matrix sum in Neural network's cost function

Hi Jordi,

First of all, thanks so much for the notebooks. They really help me to follow through the course.
I have one question in your notebook 4, nnCostFunction -- where J = ... np.sum((np.log(a3.T)*(y_matrix)+np.log(1-a3).T*(1-y_matrix))).

I think this does matrix multiplication --> giving 10*10 matrix (or n_label * n_label). This gives a matrix, let's name this cost-matrix, Jc. This Jc matrix contains not only how a set of predicted values for one label differs from it's corresponding target (diagonal elements), but also how it is differs from targets of other labels (off-diagonal elements). For example, the multiplication would multiply a column of predicted values np.log(a3.T) of one label (e.g. k) with all columns of targets.

Then the code sums all elements of this matrix. This seems to over-calculate J. Instead of summing all the elements, I think only the diagonal elements are needed.

Please use this picture to accommodate my description, which might be confusing.

Please let me know if I misunderstood the code.

Best regards and thanks again,
-Tua