afcarl / ml-mnist

My solution to Machine Learning MNIST challenge 2016-2017 [winner]

ML MNIST Challenge

This contest was offered within TU Munich's course Machine Learning (IN2064).
The goal was to implement k-NN, Neural Network, Logistic Regression and Gaussian Process Classifier in python from scratch and achieve minimal average test error among these classifiers on well-known MNIST dataset, without ensemble learning.

Results

Algorithm	Description	Test Error, %
k-NN	3-NN, Euclidean distance, uniform weights. Preprocessing: Feature vectors extracted from NN.	1.13
k-NN2	3-NN, Euclidean distance, uniform weights. Preprocessing: Augment (training) data (×9) by using random rotations, shifts, Gaussian blur and dropout pixels; PCA-35 whitening and multiplying each feature vector by e11.6 · s, where s – normalized explained variance by the respective principal axis. (equivalent to applying PCA whitening with accordingly weighted Euclidean distance).	2.06
NN	MLP 784-1337-D(0.05)-911-D(0.1)-666-333-128-10 (D – dropout); hidden activations – LeakyReLU(0.01), output – softmax; loss – categorical cross-entropy; 1024 batches; 42 epochs; optimizer – Adam (learning rate 5 · 10–5, rest – defaults from paper). Preprocessing: Augment (training) data (×5) by using random rotations, shifts, Gaussian blur.	1.04
LogReg	32 batches; 91 epoch; L2-penalty, λ = 3.16 · 10–4; optimizer – Adam (learning rate 10–3, rest – defaults from paper) Preprocessing: Feature vectors extracted from NN.	1.01
GPC	794 random data points were used for training; σn = 0; RBF kernel (σf = 0.4217, γ = 1/2l2 = 0.0008511); Newton iterations for Laplace approximation till ΔLog-Marginal-Likelihood ≤ 10–7; solve linear systems iteratively using CG with 10–7 tolerance; for prediction generate 2000 samples for each test point. Preprocessing: Feature vectors extracted from NN.	1.59

Visualizations

And more available in experiments/plots/.

How to install

git clone https://github.com/monsta-hd/ml-mnist
cd ml-mnist/
pip install -r requirements.txt

After installation, tests can be run with:

make test

How to run

Check main.py to reproduce training and testing the final models:

usage: main.py [-h] [--load-nn] model

positional arguments:
  model       which model to run, {'gp', 'knn', 'knn-without-nn', 'logreg',
              'nn'}

optional arguments:
  -h, --help  show this help message and exit
  --load-nn   whether to use pretrained neural network, ignored if 'knn-
              without-nn' is used (default: False)

Experiments

Check also this notebook to see what I've tried.
Note: the approach RBM + LogReg gave only at most 91.8% test accuracy since RBM takes too long to train with given pure python code, thus it was only trained on small subset of data (and still underfitted). However, with properly trained RBM on the whole training set, this approach can give 1.83% test error (see my Boltzmann machines project)

Features

• Apart from specified algorithms, there are also PCA and RBM implementations
• Most of the classes contain doctests so they are easy to understand
• All randomness in algorithms or functions is reproducible (seeds)
• Support of simple readable serialization (JSON)
• There are also some infrastructure for model selection, feature selection, data augmentation, metrics, plots etc.)
• Support for MNIST or Fashion MNIST (both have the same structure thus both can be loaded using the same routine), haven't tried the latter yet, though

System

All computations and time measurements were made on laptop i7-5500U CPU @ 2.40GHz x 4 12GB RAM

Possible future work

Here the list of what can also be tried regarding these particular 4 ML algorithms (didn't have time to check it, or it was forbidden by the rules, e.g. ensemble learning):

• Model averaging for k-NN: train a group of k-NNs with different parameter k (say, 2, 4, ..., 128) and average their predictions;
• More sophisticated metrics (say, from scipy.spatial.distance) for k-NN;
• Weighting metrics according to some other functions of explained variance from PCA;
• NCA;
• Different kernels or compound kernels for k-NN;
• Commitee of MLPs, CNN, commitee of CNNs or more advanced NNs;
• Unsupervised pretraining for MLP/CNN;
• Different kernels or compound kernels for GPCs;
• 10 one-vs-rest GPCs;
• Use derivatives of Log-Marginal-Likelihood for multiclass Laplace approximation w.r.t kernel parameters for more efficient gradient-based optimization;
• Model averaging for GPCs: train a collection of GPCs on different parts of the data and then average their predictions (or bagging);
• IVM.