lucasxlu / comboloss

Official PyTorch Implementation for Paper <ComboLoss for Facial Attractiveness Analysis with Squeeze-and-Excitation Networks> (State-of-the-art Performance on 3 Popular Benchmark Dataset)

ComboLoss for Facial Attractiveness Analysis with Squeeze-and-Excitation Networks

Introduction

This repository holds the official PyTorch implementation of paper ComboLoss for Facial Attractiveness Analysis with Squeeze-and-Excitation Networks. With SEResNeXt50 as backbone, ComboLoss achieves state-of-the-art performance on SCUT-FBP, HotOrNot and SCUT-FBP5500 dataset, which outperforms many methods published at IJCAI, IEEE Transactions on Affective Computing, ICIP, ICASSP, ICPR, PCM and etc.

If you find the code helps your research, please cite this project as:

@article{xu2020comboloss,
  title={ComboLoss for Facial Attractiveness Analysis with Squeeze-and-Excitation Networks},
  author={Xu, Lu and Xiang, Jinhai},
  journal={arXiv preprint arXiv:2010.10721},
  year={2020}
}

Pretrained Models on SCUT-FBP5500 with 60%/40% data splitting setting: ComboLoss_SCUT-FBP5500. We also provide inference.py code.

Data Description

Dataset	Median	Mean
SCUT-FBP	2.549	2.694
HotOrNot	0.0369	0.0039
SCUT-FBP5500	3	2.99

Performance Evaluation

Evaluation & Ablation Analysis on SCUT-FBP5500 (6/4 splitting strategy)

Backbone	Loss	MAE	RMSE	PC
SEResNeXt50	L1	0.2212	0.2941	0.9012
SEResNeXt50	MSE	0.2189	0.2907	0.9041
SEResNeXt50	SmoothL1	0.2204	0.2901	0.9050
ComboNet (SEResNeXt50)	CombinedLoss (alpha=1, beta=1, gamma=1)	0.2135	0.2818	0.9099
ComboNet (SEResNeXt50)	CombinedLoss (alpha=2, beta=1, gamma=1)	0.2191	0.2891	0.9066
ComboNet (SEResNeXt50)	CombinedLoss (alpha=2, beta=1, gamma=1)	0.2126	0.2813	0.9117
ComboNet (SEResNeXt50)	CombinedLoss (alpha=3, beta=1, gamma=1)	0.2190	0.2894	0.9053
ComboNet (SEResNeXt50)	CombinedLoss (alpha=1, beta=2, gamma=1)	0.2150	0.2868	0.9063
ComboNet (SEResNeXt50)	CombinedLoss (alpha=1, beta=2, gamma=1)	0.2176	0.2895	0.9044
ComboNet (SEResNeXt50)	CombinedLoss (alpha=1, beta=3, gamma=1)	0.2171	0.2862	0.9071
ComboNet (ResNet18)	CombinedLoss (alpha=1, beta=1, gamma=1)	0.2215	0.2936	0.9021
ComboNet (ResNet18)	CombinedLoss (alpha=1, beta=2, gamma=1)	0.2202	0.2907	0.9041
ComboNet (ResNet18)	CombinedLoss (alpha=1, beta=3, gamma=1)	0.2252	0.2991	0.8980
ComboNet (ResNet18)	CombinedLoss (alpha=2, beta=1, gamma=1)	0.2557	0.3362	0.8780
ComboNet (ResNet18)	CombinedLoss (alpha=3, beta=1, gamma=1)	0.2513	0.3364	0.8788

Evaluation on SCUT-FBP

Backbone	CV	MAE	RMSE	PC
ComboNet (SEResNeXt50)	1	0.2689	0.3340	0.9144
ComboNet (SEResNeXt50)	2	0.2456	0.3050	0.9063
ComboNet (SEResNeXt50)	3	0.2436	0.3095	0.9082
ComboNet (SEResNeXt50)	4	0.2282	0.2992	0.9238
ComboNet (SEResNeXt50)	5	0.2171	0.2889	0.9051
ComboNet (SEResNeXt50)	AVG	0.2441	0.3122	0.9090

Evaluation on HotOrNot

Backbone	CV	MAE	RMSE	PC
ComboNet (SEResNeXt50)	1	0.8207	1.0379	0.5168
ComboNet (SEResNeXt50)	2	0.8273	1.0552	0.5004
ComboNet (SEResNeXt50)	3	0.8223	1.0399	0.5148
ComboNet (SEResNeXt50)	4	0.8108	1.0241	0.5080
ComboNet (SEResNeXt50)	5	0.8256	1.0487	0.4747
ComboNet (SEResNeXt50)	AVG	0.8213	1.0412	0.5029

Evaluation on SCUT-FBP5500 (5-Fold Cross Validation)

Backbone	CV	MAE	RMSE	PC
ComboNet (SEResNeXt50)	1	0.2119	0.2751	0.9157
ComboNet (SEResNeXt50)	2	0.2084	0.2751	0.9164
ComboNet (SEResNeXt50)	3	0.1998	0.2711	0.9215
ComboNet (SEResNeXt50)	4	0.2050	0.2693	0.9208
ComboNet (SEResNeXt50)	5	0.1999	0.2615	0.9250
ComboNet (SEResNeXt50)	AVG	0.2050	0.2704	0.9199

Comparison with prior arts on SCUT-FBP5500

Models	Published At	MAE	RMSE	PC
ResNeXt-50	CVPR'16	0.2291	0.3017	0.8997
ResNet-18	CVPR'16	0.2419	0.3166	0.8900
AlexNet	NIPS'12	0.2651	0.3481	0.8634
HMTNet	ICIP'19	0.2380	0.3141	0.8912
AaNet	IJCAI'19	0.2236	0.2954	0.9055
R^2 ResNeXt	ICPR'18	0.2416	0.3046	0.8957
R^3CNN	IEEE Trans on Affective Computing	0.2120	0.2800	0.9142
ComboLoss (Ours)	-	0.2050	0.2704	0.9199

Ablation Study (6/4 splitting strategy)

Model	w/wo balanced Xent Loss	MAE	RMSE	PC
SEResNeXt50 + ComboLoss	w	0.2126	0.2813	0.9117
SEResNeXt50 + ComboLoss	wo	0.2115	0.2814	0.9099

Reference

1. Xu L, Xiang J, Yuan X. CRNet: Classification and Regression Neural Network for Facial Beauty Prediction[C]//Pacific Rim Conference on Multimedia. Springer, Cham, 2018: 661-671.
2. Lin L, Liang L, Jin L, et al. Attribute-aware convolutional neural networks for facial beauty prediction[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press, 2019: 847-853.
3. Xu L, Fan H, Xiang J. Hierarchical Multi-Task Network For Race, Gender and Facial Attractiveness Recognition[C]//2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019: 3861-3865.
4. Liu X, Li T, Peng H, et al. Understanding beauty via deep facial features[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2019: 0-0.
5. Liang L, Lin L, Jin L, et al. SCUT-FBP5500: A diverse benchmark dataset for multi-paradigm facial beauty prediction[C]//2018 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018: 1598-1603.
6. Lin L, Liang L, Jin L. Regression Guided by Relative Ranking Using Convolutional Neural Network (R3CNN) for Facial Beauty Prediction[J]. IEEE Transactions on Affective Computing, 2019.
7. Lin L, Liang L, Jin L. R 2-ResNeXt: A ResNeXt-Based Regression Model with Relative Ranking for Facial Beauty Prediction[C]//2018 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018: 85-90.

comboloss's Issues

How to test

Hi,
thanks for sharing your work.
In your code, I can find the training code and the pre-trained model, but I wonder if you can kindly share the test code that uses your pre-trained model.

Thanks in advance. BR.

[NVIDIA] [CUDA] Warning: NVIDIA GeForce RTX Laptop GPU with CUDA capability sm_86 is not compatible

Hi, I got this warning

NVIDIA GeForce RTX 3080 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3080 Laptop GPU GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

My compute capability: 8.6

Fix to this:

python3.7 -m venv .venv
source .venv/bin/activate
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Validate that it fixed:

$ python3
Python 3.7.12 (default, Mar 11 2022, 12:03:27) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
t>>> torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
>>> exit()

Hope this helps you!

used inference.py in docker have error

hi lucasxlu
Thank you very much for posting the code on GitHub
I used your code in Docker but I got an error while loading the model
Can you guide me on this
The error text is as follows

Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.7/site-packages/django/apps/config.py", line 244, in create
web_1 | app_module = import_module(app_name)
web_1 | File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module
web_1 | return _bootstrap._gcd_import(name[level:], package, level)
web_1 | File "", line 1006, in _gcd_import
web_1 | File "", line 983, in _find_and_load
web_1 | File "", line 953, in _find_and_load_unlocked
web_1 | File "", line 219, in _call_with_frames_removed
web_1 | File "", line 1006, in _gcd_import
web_1 | File "", line 983, in _find_and_load
web_1 | File "", line 965, in _find_and_load_unlocked
web_1 | ModuleNotFoundError: No module named 'classification_branch'
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "manage.py", line 22, in
web_1 | main()
web_1 | File "manage.py", line 18, in main
web_1 | execute_from_command_line(sys.argv)
web_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
web_1 | utility.execute()
web_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/init.py", line 395, in execute
web_1 | django.setup()
web_1 | File "/usr/local/lib/python3.7/site-packages/django/init.py", line 24, in setup
web_1 | apps.populate(settings.INSTALLED_APPS)
web_1 | File "/usr/local/lib/python3.7/site-packages/django/apps/registry.py", line 91, in populate
web_1 | app_config = AppConfig.create(entry)
web_1 | File "/usr/local/lib/python3.7/site-packages/django/apps/config.py", line 250, in create
web_1 | app_config_class.qualname,
web_1 | django.core.exceptions.ImproperlyConfigured: Cannot import 'classification_branch.bias'. Check that 'process.apps.ProcessConfig.name' is correct.

what's the difference of these two result

ComboNet (SEResNeXt50) | CombinedLoss (alpha=2, beta=1, gamma=1) | 0.2191 | 0.2891 | 0.9066
ComboNet (SEResNeXt50) | CombinedLoss (alpha=2, beta=1, gamma=1) | 0.2126 | 0.2813 | 0.9117

same config, two different result
because of randomness?