The test set is being used as validation set about pytorch-cifar

[zjysteven]

It is not about batchnorm statistics... It is just that evaluating on the test set to select the best model (e.g., best checkpoint and hyperparameters) goes against the basic practice/assumption of machine learning and is not realistic. In real world, there is no way to obtain the expected test samples before the model is deployed.

[wihn2021]

But the net is set to eval mode before being tested, while it is set to train mode before training, using:
net.train() and net.eval()

[zjysteven]

Agreed. Even more concerning, many papers now are reporting their performance using the best results on the test set.

[wihn2021]

🤔right.✌️

[melhzy]

Yes. It is a big issue here though. The test set is been used as the validation set. That means the models trained in the framework memorize the data patterns from the test set and train set. Overall, it causes overfitting.

[yolunghiu]

您好：    您的邮件我已收到，我会尽快回复。                                                                                       刘洪宇