tsitsimis / from-scratch

Machine Learning implementations from scratch. Using minimal dependencies this collection intends to cover fundamental machine learning algorithms: from linear regression to neural networks

From Scratch

A collection of Machine Learning algorithms implemented in Python from scratch.

Motivation

Transforming an algorithm from math equations to code is a sometimes difficult but always fruitful journey every Data Scientist should take at least once. The goal of this project is to go down to the last detail of each algorithm, fill the gap between math and code, demystify ML models complexity and revisit their mathematical background.

Implementations

• Supervised Learning

  • Multi-layer Neural Networks [demo] [code]
  • Classification Trees [demo] [code]
  • Adaptive Boosting [demo] [code]
  • Support Vector Machine [demo] [code]
  • Linear Regression [demo] [code]

• Unsupervised Learning

  • K-means [demo] [code]
  • DBSCAN [demo] [code]
  • Principal Component Analysis (PCA) [demo] [code]

• Reinforcement Learning

  • N-armed Bandits [demo] [code]
  • Dynamic Programming [demo] [code]
  • SARSA [demo] [code]
  • Q-Learning [demo] [code]

Dependencies

• numpy: Used in all implementations for vector/matrix operations and vectorized calculations
• cvxopt: Used in SVM for solving the quadratic programming problem
• scipy: Borrowed its KDTree implementation for fast nearest neighbours calculation

Demos

Each algorithm is accompanied with a notebook with mathematical background, application of the methodology on toy datasets and visualizations.

Supervised Learning

Multi-layer Neural Networks

Fead-forward Neural Network
---------------------------
4 layers: [10, 10, 10, 1]

       Inputs
       \ | /
 o o o o o o o o o o 
       \ | /
 o o o o o o o o o o 
       \ | /
 o o o o o o o o o o 
          | 
          o 
       Outputs

Classification Trees

{'expr': 'feature_2 <= 2.45',
 'no': {'expr': 'feature_3 <= 1.75',
        'no': {'expr': 'feature_2 <= 4.85',
               'no': {'label': 2.0},
               'yes': {'label': 2.0}},
        'yes': {'expr': 'feature_2 <= 4.95',
                'no': {'label': 2.0},
                'yes': {'label': 1.0}}},
 'yes': {'expr': 'feature_0 <= 4.35',
         'no': {'expr': 'feature_0 <= 4.45',
                'no': {'label': 0.0},
                'yes': {'label': 0.0}},
         'yes': {'label': 0.0}}}

Support Vector Machine

Linear Regression

Unsupervised Learning

K-means

DBSCAN

Principal Component Analysis (PCA)

Reinforcement Learning

N-armed Bandits

Dynamic Programming

Q-Learning

SARSA