uuujf / sgdnoise Goto Github PK

Hi,

Recently I am following your work The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects and have been doing an experiment similar to 5.3. FashionMNIST with corrupted labels.

I am planning to reproduce this work. At beginning, I programmed a PyTorch version of this experiment following the training details in D.3. FashionMNIST with corrupted labels. But I cannot reproduce the result in Fig.1. The result given by my code is shown in Fig. 2. In this part, I have two questions:

1. The test accuracy keeps at around 0.70 (refer to line GD pre 0.1 in Fig.2) after pretraining by GD, which is far higher than that in Fig.1 (0.6622).

2. Test accuracy is much lower than GD after keeping training by SGD (refer to line MSGD 0.07 20 in Fig.2). It does not grow up but drops a lot instead, which is against to Fig.1.

I run my code several times but unfortunately above questions keep showing up.

(Fig. 1)

(Fig. 2)

Later on, I found this repo. But I still have a few questions.

1. In the paper, 1200 samples in total are used to train but in this code it generates 6000 samples in training dataset(Fig.3). Please refer to line 80(click here to github website). I think it is just a typo and change it to 1200.

2. I ran gd.py and sgd.py after I change it to 1200. I tried several times but I still meet the problem above as my own code(Fig.4).

(Fig.3)

(Fig.4)

I really appreciate your insightful work and innovative idea. But now I really have no idea about this experiment. Would you please give me some advice? Thank you a lot for your help and I am looking forward to hearing from you soon!

Best wishes!