99991 / double-descent

This is an (unsuccessful) attempt at testing double descent without changing the training regime

Repository description

This is an (unsuccessful) attempt at testing double descent without changing the training regime, in particular Figure 4 from the paper "Reconciling modern machine learning practice and the bias-variance trade-off".

Double descent

One might expect that the test loss of a model decreases with increasing model capacity. However, some people observed the so-called "double descent phenomenon" where the test loss increases for a shot while, just to fall off again with even larger model capacity.

The squared loss should look like Figure 4 from paper with change in training regime:

Or like Figure 9 (c) from paper without weight reuse:

Results

Unfortunately, no double descent can be observed.

Usage

1. Install PyTorch, torchvision and matplotlib
2. Run python double_descent_mnist.py to generate log.json.
3. Run plot_log.py to generate double_descent.png and to show the resulting plot.