This is an (unsuccessful) attempt at testing double descent without changing the training regime, in particular Figure 4 from the paper "Reconciling modern machine learning practice and the bias-variance trade-off".
One might expect that the test loss of a model decreases with increasing model capacity. However, some people observed the so-called "double descent phenomenon" where the test loss increases for a shot while, just to fall off again with even larger model capacity.
The squared loss should look like Figure 4 from paper with change in training regime:
Or like Figure 9 (c) from paper without weight reuse:
Unfortunately, no double descent can be observed.
- Install PyTorch, torchvision and matplotlib
- Run
python double_descent_mnist.py
to generatelog.json
. - Run
plot_log.py
to generatedouble_descent.png
and to show the resulting plot.