msuzen / bristol

Parallel random matrix tools and complexity for deep learning

Bristol

Parallel random matrix tools and random matrix theory deep learning applications. Generate matrices from Circular Unitary Ensemble (CUE), Circular Ortogonal Ensemble (COE) and Circular Symplectic Ensemble (CSE). Additional spectral analysis utilities are also implemented, such as computation of spectral density and spectral ergodicity for complexity of deep learning architectures.

Features

• Generation of Circular Ensembles: CUE, COE and CSE.
• Random matrices: Reproducibility both in serial and parallel processing.
• Eigenvalue Spectra, spectral densitiy.
• Kullbach-Leibler divergence and spectral ergodicity measure functionality.
• Cascading Periodic Spectral Ergodicity (cPSE) : This is a complexity measure and could also detect when to stop addinig more layers.

Installation

Install with pip from pypi, as of 0.2.14 we prefer Python > 3.9

pip install bristol

Running tests

bash run_tests.py

To use the latest development version

pip install -upgrade git+https://github.com/msuzen/bristol.git

Documentation

Complexity of a deep learning model: cPSE

Vanilla case

In the vanilla case a list of matrices that are representative of ordered set of weight matrices can be used to compute cPSE over layers. As an examples:

from bristol import cPSE
import numpy as np
np.random.seed(42)
matrices = [np.random.normal(size=(64,64)) for _ in range(10)]
(d_layers, cpse) = cPSE.cpse_measure_vanilla(matrices)

Even for set of Gaussian matrices, d_layers decrease. Note that different layer types should be converted to a matrix format, i.e., CNNs to 2D matrices. See the main paper.

When should I stop adding more layers in deep learning?

d_layers is a decreasing vector, it will saturate at some point, that point is where adding more layers won’t improve the performance. This is data, learning and architecture independent measure.

For torch models

You need to put your model as pretrained model format of PyTorch. An example for vgg, and use cPSE.cpse_measure function simply:

from bristol import cPSE
import torchvision.models as models
netname = 'vgg11'
pmodel = getattr(models, netname)(pretrained=True)
(d_layers, cpse) = cPSE.cpse_measure(pmodel)

This would give cpse a single number expressing the complexity of your network and d_layers evolution of periodic spectral ergodicity withing layers as a vector, order matters.

Random Stream Chunking

Package employs a technique called random stream chunking to ensure reproducibility
between parallel and serial runs. A blog post explaining this:
Exact reproducibility of stochastic simulations for parallel and serial algorithms simultaneously Random Stream Chunking

Prototype notebooks

• Basics of circular ensembles ipynb.

• Computing spectral ergodicity for generated matrices ipynb. This is to reproduce the main figure from arXiv:1704.08693.

• The concept of cascading periodic ergodicity (cPSE) ipynb This is only to reproduce paper's results from arXiv:1911.07831.

• Empirical deviations of semicircle law in mixed-matrix ensembles,
  M. Suezen, hal-03464130 | ipynb Reproduces the work with the same title.

Contact

• Please create an issue for any type of questions or contact msuzen.

References

• Berry, M V & Pragya Shukla 2013, Hearing random matrices and random waves, New. J. Phys. 15 013026 (11pp) berry456

• Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices, Mehmet Süzen, Cornelius Weber, Joan J. Cerdà, arXiv:1704.08693

• Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search, Mehmet Süzen, Cornelius Weber, Joan J. Cerdà, arXiv:1911.07831

• Empirical deviations of semicircle law in mixed-matrix ensembles,
  M. Suezen, hal-03464130 | ipynb Reproduces the work with the same title.

Citation

If you use the ideas or tools from this package please do cite our manuscripts.

@article{suezen2021a,
    title={Empirical deviations of semicircle law in mixed-matrix ensembles},
    author={Mehmet Süzen},
    year={2021},
    eprint={hal-03464130},
    url={https://hal.archives-ouvertes.fr/hal-03464130}
}

@article{suezen2019a,
    title={Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search},
    author={Mehmet Süzen and Joan J. Cerdà and Cornelius Weber},
    year={2019},
    eprint={1911.07831},
    archivePrefix={arXiv},
    primaryClass={stat.ML}
}

@article{suezen2017a,
    title={Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices},
    author={Mehmet Süzen and Cornelius Weber and Joan J. Cerdà},
    year={2017},
    eprint={1704.08303},
    archivePrefix={arXiv},
    primaryClass={stat.ML}
}

bristol's Issues

drop names within code

Drop author names within code as contributors covers this.

Gen data module

Example data set generation module, for example see code here. Note that this is older code, so Circular module name capitalized.

fix CI

CI is broken as travis org is ceased to exist.

Drop python 2 and <3.7 dependency

Referbish.

gen tool in pytorch.

ergodicity module

introduce ergodicitiy module, using functions from the notebook.

Mixed Matrix ensemble generator with CUE

As defined in
Equivalence in Deep Neural Networks via Conjugate Matrix Ensembles and
notebook

Mixed matrix ensembles conjugacy and equivalence tools

As defined in
Equivalence in Deep Neural Networks via Conjugate Matrix Ensembles and
notebook

Gaussian parameters

Introduce mean, variance in matrix generation.

cPSE monitoring during training epochs: Investigate

Introduce cPSE in PyTorch monitor during training epochs.

type hints

moving to static type hint:
https://docs.python.org/3/library/typing.html

Add unit tests for circular

Circular module needs unit tests.

Density argument for TM/KL metric

Introduce density argument for KL/TM metric

Tutorial documentation

Detailed doc directory using sphyx with implementation details.

Support for Gaussian ensembles

Gaussian orthogonal ensemble (GOE), Gaussian unitary ensemble (GUE) and Gaussian symplectic ensemble (GSE)

Spectral density: handle real only list

Parallel matrix generation with theoretical test/correctness cases.

Eigenvalues can be generated in parallel but not single matrices.

cPSE measure

add cPSE measure

Repeated Computation of CUE

When computing COE, and CSE, provide an option to use already existing CUE.

Update method documentation

Use docstring compatible structure, such as the following taken from scipy:

def angle(z, deg=0):
    """
    Return the angle of the complex argument.

    Parameters
    ----------
    z : array_like
        A complex number or sequence of complex numbers.
    deg : bool, optional
        Return angle in degrees if True, radians if False (default).

    Returns
    -------
    angle : ndarray or scalar
        The counterclockwise angle from the positive real axis on
        the complex plane, with dtype as numpy.float64.

    See Also
    --------
    arctan2
    absolute



    Examples
    --------
    >>> np.angle([1.0, 1.0j, 1+1j])               # in radians
    array([ 0.        ,  1.57079633,  0.78539816])
    >>> np.angle(1+1j, deg=True)                  # in degrees
    45.0

    """
    if deg:
        fact = 180/pi
    else:
        fact = 1.0

cPSE pytorch with state dict.

state dict can be used to produce cPSE.

investigate for tensorflow models support

Along with internal representation

Update ergodicity_random_matrix_17a.ipynb

Update python notebook after gen_module as ergodicity related methods moved to ergodicity module.

Close pool in multiprocess.

Close and Join the child processes.

Fix url from the mixed ensembles notebook

https://github.com/msuzen/bristol/blob/master/works/mixedMatrixEnsembles/deviation_semicirclelaw.ipynb

cPSE: model independent version.

Check unit tests for absolute difference

Some unit tests might be logically wrong

Circular ensemble random numbers

Introduce random number generation utils for circular ensembles. Possibly merge with #13

General review core code.

introduce Manifest file

Introduce MANIFEST.in

include README.md
include LICENSE.txt

Update description

Parallel random matrix tools and complexity for deep learning. Drop other clutter from the description.

Visualise random matrix

Given generated matrix, visualise in colour mode:

• color/bw
• historgram

Store the object. Preferable use matplotlib.