yukkyo / pytorch-filterresponsenormalizationlayer

PyTorch implementation of "Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks"

PyTorch-Filter Response Normalization Layer(FRN)

PyTorch implementation of Filter Response Normalization Layer(FRN)

[1911.09737] Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks

0. How to apply FRN to your model

Replace BatchNorm2d + ReLU in the model with FRN + TLU yourself. Currently, it is difficult to easily replace them with functions. Because many models use the same ReLU in various places.

1. Experiment(Classification)

We use Best Artworks of All Time | Kaggle dataset. This dataset contains 49 artists and their pictures.
In this experiment, we classify artist by picture.

1.0 Assumed libraries

• torch==1.3.1
• catalyst==19.11.6
• albumentations==0.4.3
• NVIDIA/apex
  • If you use --fp16 option

1.1 Get dataset

If you can use kaggle API command, you can download easily

$ cd input
$ kaggle datasets download -d ikarus777/best-artworks-of-all-time
$ unzip best-artworks-of-all-time.zip -d artworks

Or download directly from Best Artworks of All Time | Kaggle

I assume the following directory structure.

input
├── artworks
│   ├── artists.csv
│   ├── images
│   │   └── images
│   │       ├── Alfred_Sisley
│   │       │   ├── Alfred_Sisley_1.jpg
│   │       │   ├── Alfred_Sisley_10.jpg
│   │       │   ├── ...

1.2 Train(and Valid)

You can use --fp16 if you installed nvidia/apex. But FRN is not tuned for FP16, you should turn off --fp16 when use --frn.

$ python train_cls.py --fp16
$ python train_cls.py --frn

pytorch-filterresponsenormalizationlayer's Issues

bnrelu_to_frn is not work now !!!

bnrelu_to_frn is not work if definition order is different from forward order.

tau and epsilon

Hello, thanks for your work.
I have read your codes and found something mismatching with original paper.
The first is the 'tau' in TLU. In paper, it's said that tau is a vector of size num_channels. But in your codes, It's just a single value.
The second is the epsilon. In paper, it's said when using in conv, epsilon is a constant value of 10^-6. but in your codes , it's a learnable parameter.
Will these mismatches hurt the preformance?

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Not sure why this occurs.

Setting torch.backends.cudnn.enabled=False fixes this issue. Unfortunately it also increases memory consumption.

senet_frn first stage bn or frn?

https://github.com/yukkyo/PyTorch-FilterResponseNormalizationLayer/blob/master/senet_frn.py#L287

        if input_3x3:
            layer0_modules = [
                ('conv1', nn.Conv2d(3, 64, 3, stride=2, padding=1,
                                    bias=False)),
                ('bn1', nn.BatchNorm2d(64)),
                ('relu1', nn.ReLU(inplace=True)),
                ('conv2', nn.Conv2d(64, 64, 3, stride=1, padding=1,
                                    bias=False)),
                ('bn2', nn.BatchNorm2d(64)),
                ('relu2', nn.ReLU(inplace=True)),
                ('conv3', nn.Conv2d(64, inplanes, 3, stride=1, padding=1,
                                    bias=False)),
                ('bn3', nn.BatchNorm2d(inplanes)),
                ('relu3', nn.ReLU(inplace=True)),
            ]
        else:
            layer0_modules = [
                ('conv1', nn.Conv2d(3, inplanes, kernel_size=7, stride=2,
                                    padding=3, bias=False)),
                ('bn1', nn.BatchNorm2d(inplanes)),
                ('relu1', nn.ReLU(inplace=True)),
            ]
        # To preserve compatibility with Caffe weights `ceil_mode=True`
        # is used instead of `padding=1`.
        layer0_modules.append(('pool', nn.MaxPool2d(3, stride=2,
                                                    ceil_mode=True)))

Sorry but I'm a little confused. When using FRN in senet, should all batch norms be replaced by FRN?