janpf / self-supervised-multi-task-aesthetic-pretraining

Code and data for "Self-Supervised Multi-Task Pretraining Improves Image Aesthetic Assessment"

Self-Supervised Multi-Task Pretraining Improves Image Aesthetic Assessment

In this repository you find the code and data for the paper "Self-Supervised Multi-Task Pretraining Improves Image Aesthetic Assessment" published at NTIRE21:

Neural networks for Image Aesthetic Assessment are usually initialized with weights of pretrained ImageNet models and then trained using a labeled image aesthetics dataset. We argue that the ImageNet classification task is not well-suited for pretraining, since content based classification is designed to make the model invariant to features that strongly influence the image's aesthetics, e.g. style-based features such as brightness or contrast. We propose to use self-supervised aesthetic-aware pretext tasks that let the network learn aesthetically relevant features, based on the observation that distorting aesthetic images with image filters usually reduces their appeal. To ensure that images are not accidentally improved when filters are applied, we introduce a large dataset comprised of highly aesthetic images as the starting point for the distortions. The network is then trained to rank less distorted images higher than their more distorted counterparts. To exploit effects of multiple different objectives, we also embed this task into a multi-task setting by adding either a self-supervised classification or regression task. In our experiments, we show that our pretraining improves performance over the ImageNet initialization and reduces the number of epochs until convergence by up to 47%. Additionally, we can match the performance of an ImageNet-initialized model while reducing the labeled training data by 20%. We make our code, data, and pretrained models available.

Underlying code

The code in this repository is based on / uses:

• github.com/kentsyx/Neural-IMage-Assessment
• github.com/spijkervet/SimCLR
• github.com/gidariss/FeatureLearningRotNet

Files

https://oc.informatik.uni-wuerzburg.de/s/6L5CNQsbp2yMpwC

PW: oosCmnMQ10

Citation

@InProceedings{Pfister_2021_CVPR,
    author    = {Pfister, Jan and Kobs, Konstantin and Hotho, Andreas},
    title     = {Self-Supervised Multi-Task Pretraining Improves Image Aesthetic Assessment},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2021},
    pages     = {816-825}
}

self-supervised-multi-task-aesthetic-pretraining's Issues

Hello, the pre-trained model file cannot be found

data set

Hello, can you provide the data set?

Evaluation of LCC, SRCC, ACC, MAE

Hi, very nice work.
Could you provide the code of evaluation (LCC, SRCC, ACC, MAE)?

error: Xmp schema version 3 in edit.xmp not supported

WARNING: either your user id or the effective user id are 0. are you running darktable as root?
[defaults] found a 64-bit system with 527873464 kb ram and 72 cores (0 atom based)
[defaults] setting very high quality defaults
error: Xmp schema version 3 in edit.xmp not supported
error: can't open xmp file edit.xmp

Could you share the GoogleDrive link of the original pexels dataset?

First of all, thank you very much for making your excellent work available! It's very beneficial to me!
Could you share the GoogleDrive link of the original pexels dataset?
Because even if I use the shellscript(“download_pexels.sh”) you provide, the download speed is still slow. And I don't know the size of the dataset and whether my disk space is enough.
So I really hope to get your help. Thank you very much.

Why does your public pretrained model have the same result as a model that trains from scratch?

Your method doesn't improve NIMA at all. Can you explain this problem?

Incomplete program

There is no ‘SimCLR’ and 'RotNet' in 'relatedWorks' package, but you referenced the code in 'relatedWorks/tune_related/NIMA.py'

Downloading Pexels.com Dataset

Thank you for your great work and open source code. Can you provide the pre-training dataset of pexels you collected? Thanks a lot.

How to test?

Hello! Could you provide some instructions of how to train and test your solution?
Also pretrained model file is not available (link not active).
Thank you!

Can you share your pretrained models/backbones?

thanks for your work!

How to speed up darktable? It is very slow.

about 1.7s for subprocess.run(["darktable-cli", img_path, edit_file, edited_img_path, "--core", "--library", ":memory:"])