paulgavrikov / diffusion-noise-cls-pytorch

This is an unofficial implementation of the diffusion-style noise frontend in "Intriguing properties of generative classifiers" by Priyank Jaini, Kevin Clark, Robert Geirhos to improve the shape-bias of vision models.

diffusion-noise-cls-pytorch

This is an unofficial implementation of the diffusion-style noise frontend in "Intriguing properties of generative classifiers" by Priyank Jaini, Kevin Clark, Robert Geirhos to improve the shape-bias of vision models.

Training

Reproduce the training of ResNet-50 (assumes you have 4 GPUs on 1 node)

torchrun --nproc-per-node=4 train.py --amp --no-normalization --diffusion-noise

If you have a different number of GPUs, adjust the batch size (via --batch-size B) such that B * GPUs = 256.

Please note that the resulting model is not normalized by channel means/stds, i.e., you must ensure that input data is in the [0, 1] range.

Results

Model	ImageNet-val	Shape-Bias	Mean OOD
RN-50 (90 Epoch) - eval with noise	51.64 %	0.73	46.63 %
RN-50 (90 Epoch) - eval w/o noise	67.22 %	0.51	56.06 %

The checkpoint is available on HuggingFace.

Please note that the evaluation with noise will give you slightly different results based due to the non-deterministic nature of noise.

Differences to original publication/codebase

The original codebase is written in JAX, trains for 300 epochs and does not use AMP - as such the results do not fully align.

Credits

Thanks to Robert Geirhos for the help in reproducing the results! The training script is based on the example in PyTorch.

diffusion-noise-cls-pytorch's Issues

Difficulty replicating shape bias result

Hi Paul,

Thanks for implementing this paper!

I trained ResNet-50 for 90 epochs with diffusion noise using your code, in an effort to replicate the results you report in your README.

I get very similar validation accuracies, but very different shape bias scores as shown in the table below. The second and third row are the diffusion-noise-trained ResNets without and with noise added during inference. As you can see my shape bias scores are much lower than yours.

To measure shape bias I'm using code from https://github.com/bethgelab/model-vs-human as follows:

from modelvshuman import Evaluate
Evaluate()(['resnet50'], ['cue-conflict'])
Evaluate()(['diffusion_resnet50'], ['cue-conflict'])
Evaluate()(['diffusion_resnet50_wdiffnoise'], ['cue-conflict'])

While testing the diffusion ResNets I'm also making sure to apply the required transforms (normalization to mean 0 and std 1, resize to 256 and center crop 224).

Any idea what might be causing my results to be so different?

Thanks,
Ajay