huyvnphan / pytorch_cifar10

Pretrained TorchVision models on CIFAR10 dataset (with weights)

PyTorch models trained on CIFAR-10 dataset

• I modified TorchVision official implementation of popular CNN models, and trained those on CIFAR-10 dataset.
• I changed number of class, filter size, stride, and padding in the the original code so that it works with CIFAR-10.
• I also share the weights of these models, so you can just load the weights and use them.
• The code is highly re-producible and readable by using PyTorch-Lightning.

Statistics of supported models

No.	Model	Val. Acc.	No. Params	Size
1	vgg11_bn	92.39%	28.150 M	108 MB
2	vgg13_bn	94.22%	28.334 M	109 MB
3	vgg16_bn	94.00%	33.647 M	129 MB
4	vgg19_bn	93.95%	38.959 M	149 MB
5	resnet18	93.07%	11.174 M	43 MB
6	resnet34	93.34%	21.282 M	82 MB
7	resnet50	93.65%	23.521 M	91 MB
8	densenet121	94.06%	6.956 M	28 MB
9	densenet161	94.07%	26.483 M	103 MB
10	densenet169	94.05%	12.493 M	49 MB
11	mobilenet_v2	93.91%	2.237 M	9 MB
12	googlenet	92.85%	5.491 M	22 MB
13	inception_v3	93.74%	21.640 M	83 MB

Details Report & Run Logs

Weight and Biases' details report for this project WandB Report

Weight and Biases' run logs for this project WandB Run Log. You can see each run hyper-parameters, training accuracy, validation accuracy, loss, time taken.

How To Cite

How to use pretrained models

Automatically download and extract the weights from Box (933 MB)

python train.py --download_weights 1

Or use Google Drive backup link (you have to download and extract manually)

Load model and run

from cifar10_models.vgg import vgg11_bn, vgg13_bn, vgg16_bn, vgg19_bn

# Untrained model
my_model = vgg11_bn()

# Pretrained model
my_model = vgg11_bn(pretrained=True)
my_model.eval() # for evaluation

If you use your own images, all models expect data to be in range [0, 1] then normalized by

mean = [0.4914, 0.4822, 0.4465]
std = [0.2471, 0.2435, 0.2616]

How to train models from scratch

Check the train.py to see all available hyper-parameter choices. To reproduce the same accuracy use the default hyper-parameters

python train.py --classifier resnet18

How to test pretrained models

python train.py --test_phase 1 --pretrained 1 --classifier resnet18

Output

{'acc/test': tensor(93.0689, device='cuda:0')}

Requirements

Just to use pretrained models

• pytorch = 1.7.0

To train & test

• pytorch = 1.7.0
• torchvision = 0.7.0
• tensorboard = 2.2.1
• pytorch-lightning = 1.1.0

pytorch_cifar10's Issues

Do you know FLOPs of every model?

your only show params of every model in the table,but i need FLOPs of every model,can you show me.thanks

Can't replicate results

Hi,

I verified the noted results of 94.07% percent of the vgg16_bn pretrained net. However, when trying to train the vgg16_bn net from scratch using the noted training parameters (minibatch 256, sgd nesterov momentum 0.9, weight decay 0.001, step learning with reduction factor 0.1 every 200 epochs, 600 epochs total) and the given normalization values, I only achieved best at epoch 332 with validation set acc 0.896300 and training set acc 1.000000. Gap of 0.103700.

How come?

AdaptiveAvgPool2d in VGG creates redundancy

In the implementation of VGG you keep the original AdaptiveAvgPool2d layer with output_size = (7, 7).
This layer creates 49 copies of the same value in the case of CIFAR10 images whose size is 32x32.

In the original VGG for ImageNet this layer has no effect because input images have size 224x224 and the size of the output of vgg.features is a tensor with shape (N, 512, 7, 7).
It is added to allow the system to be used with inputs of arbitrary sizes.

Using 32x32 images from CIFAR10, the output size of vgg.features is a tensor with shape (N, 512, 1, 1).
In this case the pooling layer repeat the single value 49 times to create a tensor with the desired shape.
The following linear layer has an enormous number of parameters because of this choice.

I think the pooling should be removed.

Embeddings from the trained model don't work.

Hey, I loaded your pretrained model for CIFAR-10 using the code given in the readme, and then simply tried performing inference on my own with the model. For some reason, I'm getting very poor accuracy metrics (11%) on the CIFAR 10 test set.

from torchvision.datasets import CIFAR10

dataset_dir = "/content/"
mean = [0.4914, 0.4822, 0.4465]
std = [0.2023, 0.1994, 0.2010]
transform_val = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean, std)])
dataset = CIFAR10(root= dataset_dir, train=False, transform=transform_val,  download = True)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=128, num_workers=4, pin_memory=True)

Then I perform simple inference with y=model(x) format to get the embedding, and I argmax the 10-dim values I get to obtain my predicted class values.

Also when I normalise, most of the values just turn out to be class 1 after I argmax. Very weird.

Am I doing something wrong here? I get 11% accuracy doing this.

32 or 16-bit FP

Hi,

The parameters are 32 or16-bit floating-point?

Softmax in the last layer

Did you use a Softmax in the very last layer? I see that last layer is a fully connected (nn.Linear).

Saving state_dict into another CNN

Hi there, I'm thinking of using this model for some school work (I'll cite of course).
I'm trying to save the trained weights from the vgg16 model into an identical vgg16 Net(nn.Module) thing (all the same layers and sizes and I've changed the layer names to match) because I need access to only one of the convolutional layers.

However, when I do this, the new model with the weights from vgg16_bn is acting like it's never been trained.

my_model = vgg16_bn(pretrained=True)
my_model.eval()
torch.save(my_model.state_dict(), 'vgg16_cifar_weights')

and then later in my "from scratch" vgg16
vgg.load_state_dict(torch.load('PytorchCifarModel/vgg16_cifar_weights_scratch3'))

Can you see what I'm doing wrong? This trained model would be so perfect for my project, I'm hoping that it'll work.

I'll put my from scratch vgg16 here as well in case there's an issue there. I know it's really long winded but it needs to be for my proejct. Thank you!!

class Net(nn.Module):
    
    def __init__(self):
        super().__init__()
        
        self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        
        
        self.conv1_1 = nn.Conv2d(3, 64, 3, 1, padding=1)
        self.batch1_1 = nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv1_2 = nn.Conv2d(64, 64, 3, 1, 1)
        self.batch1_2 = nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        
        self.conv2_1 = nn.Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch2_1 = nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv2_2 = nn.Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch2_2 = nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        
        self.conv3_1 = nn.Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch3_1 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv3_2 = nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch3_2 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv3_3 = nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch3_3 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        
        self.conv4_1 = nn.Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch4_1 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv4_2 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch4_2 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv4_3 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch4_3 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        
        self.conv5_1 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch5_1 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv5_2 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch5_2 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.conv5_3 = nn.Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        self.batch5_3 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        
        self.classifier = nn.Sequential(
            nn.Linear(512 * 1 * 1, 4096),
            # nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 10),
        )
        
        self.relu_f = nn.ReLU(True)
        
    def forward(self, x):
        x = self.conv1_1(x)
        x = self.relu_f(self.batch1_1(x))
        x = self.conv1_2(x)
        x = self.relu_f(self.batch1_2(x))
        
        x = self.maxpool(x)
        
        x = self.conv2_1(x)
        x = self.relu_f(self.batch2_1(x))
        x = self.conv2_2(x)
        x = self.relu_f(self.batch2_2(x))
        
        x = self.maxpool(x)
        
        x = self.conv3_1(x)
        x = self.relu_f(self.batch3_1(x))
        x = self.conv3_2(x)
        x = self.relu_f(self.batch3_2(x))
        x = self.conv3_2(x)
        x = self.relu_f(self.batch3_3(x))
        
        x = self.maxpool(x)
        
        x = self.conv4_1(x)
        x = self.relu_f(self.batch4_1(x))
        x = self.conv4_2(x)
        x = self.relu_f(self.batch4_2(x))
        x = self.conv4_2(x)
        x = self.relu_f(self.batch4_3(x))
        
        x = self.maxpool(x)
        
        x = self.conv5_1(x)
        x = self.relu_f(self.batch5_1(x))
        x = self.conv5_2(x)
        x = self.relu_f(self.batch5_2(x))
        x = self.conv5_3(x)
        x = self.relu_f(self.batch5_3(x))
        
        x = self.maxpool(x)
        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        
        return x

How did you get the pretrained models?

Hi,

I'm wondering how are the pre-trained models obtained? For example, when I'm loading the pre-trained model for resnet18, is this pre-trained model trained using imagenet or still cifar? Thanks!

Citation

My Lab was looking for a quick set of pretrained networks for use in numerical experiments for CIFAR10. We found this to be a convenient source. You do not have attribution information within the readme, how would you like to be cited?

wrong test accuracy reported

Hi and thanks for sharing the weights and the models. Correct me if I'm wrong but I beleive what you report as test accuracy is just the validation accuracy and not the final test accuracy of model evaluated on the untouched test set. You should have train/validt/test --> train_loader/valid_loader/test_loader but I can only see train and validation loaders and you report test accuracies based on the validation loader if I'm not mistaken?

Error: The `preds` should be probabilities, but values were detected outside of [0,1] range.

I don't see any softmax used on the output, so I get this error when testing (and training) the pretrained network.
python train.py --test_phase 1 --pretrained 1 --classifier resnet18

The `preds` should be probabilities, but values were detected outside of [0,1] range.
which is a reasonable error. :)

this is the part of code that is problematic:

PyTorch_CIFAR10/module.py

Line 44 in 7a6e1f9

accuracy = self.accuracy(predictions, labels)

what function should be used on the output of the model ?, softmax?
why is this not in your codes ?

Originally posted by @kfarivar in #12 (comment)

Hyperparameters used for training

Hey there!
First of all thanks for publishing this repo and the weights, it's really helpful!
I was wondering whether you used the default hyperparameters for training that are specified in the argparser of train.py, or whether you used different ones?
I would be particularly interested in the ones for mobilenet_v2 and VGG16, if you can share.
Thanks in advance!

Error during download weights and even training

I have this problem when I tried to test the "pre-trained models" from the master branch:

RuntimeError: Dataset not found or corrupted. You can use download=True to download it

I tried to use the version from the 2022 branch, but this version does not have a --download-weights option. I have downloaded manually the weights and extracted them into the folder, and the "corruption" message continues.

I had installed the pytorch 1.7.1 and torchvision 0.8.2, because the version 1.7.0/0.7.0 it's not currently available in pytorch repo.

Any solution for this?

Pre-trained models - low accuracy?

When loading the pretrained ResNets, I am getting very low training set accuracy (around 22% on a subset of 1000 images).

This is a sample from my code:

model = cifar10_models.resnet50(pretrained=True)

mean = np.array([0.4914, 0.4822, 0.4465])
std = np.array([0.2023, 0.1994, 0.2010])

self.normalize = transforms.Normalize(mean=mean, std=std)

self.train_loader = DataLoader(
            datasets.ImageFolder(training_dir, transform=transforms.Compose([
                transforms.RandomCrop(32, padding=4),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                self.normalize,
            ])),
            batch_size=1, shuffle=True,
            num_workers=0, pin_memory=True)
       

Can you please confirm that these weights are the right ones?

AttributeError: can't set attribute

Hi there,
How to solve this AttributeError?
The full Traceback are as follows:

Traceback (most recent call last):
File "train.py", line 82, in
main(args)
File "train.py", line 40, in main
model = CIFAR10Module(args)
File "/home/amax/PyTorch_CIFAR10/module.py", line 34, in init
self.hparams = hparams
File "/home/amax/anaconda3/envs/cifar/lib/python3.6/site-packages/torch/nn/modules/module.py", line 826, in setattr
object.setattr(self, name, value)
AttributeError: can't set attribute

std different in script and readme file

Hi,

The std in readme file and data.py file is different. Any inputs?

No resnet_orig.pt

There is no resnet_orig.pt state dict in the state_dict.zip file

wrong accuracy of resnet18

I test resnet18 on the CIFAR10 test set, the accuracy is 81.24%, but the reported acc is over 93.07%

Is my test code wrong?

The code is below:

import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

from model import *


if __name__ == '__main__':
    net = resnet18(pretrained=True, device='cuda').cuda()
    net.load_state_dict(torch.load('state_dicts/resnet18.pt'))

    mean = (0.4914, 0.4822, 0.4465)
    std = (0.2471, 0.2435, 0.2616)
    t = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean, std),
    ])
    dataset = datasets.CIFAR10(root='.', train=False, transform=t, download=True)
    dataloader = DataLoader(dataset, batch_size=10, num_workers=0, shuffle=False)
    acc = 0.
    preds = []
    labels = []
    for i, (img, label) in enumerate(dataloader):
        # if i < 10:
        if 1:
            img = img.cuda()
            pred = net(img).detach().cpu()
            pred_indices = pred.argmax(1)
            preds.append(pred_indices)
            labels.append(label.cpu())
    preds = torch.cat(preds)
    labels = torch.cat(labels)
    print(sum(preds==labels)/len(preds))