snowfallingplum / ssat

[AAAI 2022] This is the official pytorch code for SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal

SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal

This is the official pytorch code for SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal, which has been accepted by AAAI-2022. Note that only trained models and test code are provided in pytorch code.

Latest News

Our latest paper was accepted by CVPR2024 and we will make the code publicly available at this URL https://github.com/Snowfallingplum/CSD-MT as soon as possible.

Training code

We have provided the complete training code for the MindSpore version of the SSAT model.

Example

We have provided test samples and trained models, you only need to run the "test.py" file and the results will be in "./results" folder .

How to test a custom dataset

1. Prepare face parsing. Face parsing is used in this code. In our experiment, face parsing is generated by https://github.com/zllrunning/face-parsing.PyTorch.
2. Put the results of face parsing in the .\test\seg1\makeup and \test\seg1\non-makeup
3. python test.py.

Our results

.

ssat's Issues

how to get seg1?

myself no makeup images,how to get seg1?

How convert to onnx model?

如何转成onnx模型呢？

About Partial Makeup Transfer

Hi @Snowfallingplum,
I see your paper provides partial makeup transfer. Could you give me some hints how to do this in your repo code.

Thank you very much!

Result img is not stable

source img:

target img:

result img0: (good)

result img1: (weird)

Sometimes generate img1 which color is cyan...I don`t know how to solve the problem.

关于模型架构的一些想法，想探讨请教一下

就是对于模型设计，我很喜欢就是输入人脸解析这个语义分割结果这个idea，，但是感觉这边处理人脸解析信息这里有点粗暴，就是只是parse 经过 semantic_encoder 然后直接和content_encoder的人脸内容信息进行torch.cat会不会有点粗暴呢？如果是我的想法会至少进行人脸解析和人脸内容进行一个逐元素相乘的操作，我想通过类似这种逐元素相乘更好的避免一些背景的干扰会不会好一点呢？我之所以想到这个原因是因为我把pytorch你们团队给出的代码用paddle写了一份，并且参数迁移过去我实验了多次，发现就是人脸需要占据整张图片的很大一部分效果才好，就是生成出来的妆容迁移往往人物背景会进行改变和变色，其中输入图片的背景也会对于模型妆容迁移进行错误干扰。

不知道你们团队对于我说的这个人脸解析语义分割信息与人脸图片融合的想法是否有过类似的idea和实验呢？就是我不是故意找茬的，就是你们的这篇论文让我受益匪浅，想更加深入的探讨一下，不知道可不可以。

如果是需要光照信息的话，人脸应该也可以获得光照信息，并不依赖背景，我这边测试了证件照还有一个其他的照片，这里给出证件照片的不太好的效果。

Generate artifacts that appear in the final image

Thank you very much for your work and the improved generalization robustness of this algorithm, but I would still like to ask you what causes the phenomenon shown in the figure.

我觉得这里是不是代码有点小瑕疵，关于这个softmax

https://github.com/Snowfallingplum/SSAT/blob/master/networks.py class SymmetryAttention其中269到272行

        energy_ab_T = torch.bmm(fb.transpose(-2, -1), fa) * alpha
        corr_ab_T = F.softmax(energy_ab_T, dim=2)  # n*HW*C @ n*C*HW -> n*HW*HW
        # print(softmax_weights.shape, b_raw.shape)
        b_warp = torch.bmm(b_raw.view(n, raw_c, h * w), corr_ab_T)  # n*HW*1

这里面相当于b_content[b,hw,c] a_content[b,c,hw]再softmax得到attention,然后b_makeupattention。也就是说b的妆容针对于a的脸部进行一个内容扭曲，但是这个softmax应该针对的是1dim啊，因为你是b_makeup*attention，应该是a每个空间点对应的b所有空间信息进行一个softmax，对吧，这边你们团队给出的代码dim等于2逻辑上与传统的qkv self attention机制思维逻辑不一样吧

training code

Great Job! Will you release the training code and train data set later? thx

dataset

Makeup Transfer 可不可以共享下 官网失效了

Result img is not stable

source img:

target img:

result img0:

result img1:

Sometimes generate result like img1 which color seems weird...

training code

Great Job! Will you release the training code later?

Can you perform makeup migration at 1024X1024 resolution? What should you do if you want a larger resolution?

hi,how to get train code ?

It's missing a lot of training-related code.