fahadshamshad / clip2protect

[CVPR 2023] Official repository of paper titled "CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search".

Hi there 👋

• 🕘 I have held technical roles as Senior Machine Learning Engineer at OMNO.AI and as a Research Associate at Information Technology University, Lahore.
• 🥇 I did my Masters in Electrical Engineering from NUST Islamabad. I did my bachelors from Institute of Space Technology, Islamabad.
• 🌱 My research interests include Image Reconstruction, Medical Image Analysis, and Computational Imaging.
• 📫 How to reach me: https://fahadshamshad.github.io | | [email protected]

clip2protect's Issues

Could you explain how can we get the values befpre `-margin` and `+margin` in utils.clip2protect_utils.get_target?

Thanks a lot.

experiment problem

I appreciate your sharing work.However, I'm facing some challenges with the generated results when I directly input the command "python main.py --data_dir input_images --latent_path latents.pt --protected_face_dir results". The generated images do not have satisfactory visual quality. I am specifically interested in performing makeup transfer. Could you please provide guidance on how to resolve this issue?"

Identity preserving latent codes

Your work is awesome. I have one question, in the paper it is mentioned that the optimization is over some identity preserving latent codes but from the code below it seems that requires_grad=True for all these latent codes and all are going in the optimizer. so are all getting updated?

https://github.com/fahadshamshad/Clip2Protect/blob/8596a82cd04a1c85ecc6a805daa2d2710d85de96/adversarial_optimization.py#L215C18-L215C18

AttributeError: 'dict' object has no attribute 'eval'

Configurations:
I have Cuda 12.0 in local system, so I had to install latest PyTorch version to enable GPU for PyTorch.
I was able to get latent code (latent.pt), obtain the FR model and placed everything in the appropriate directories

When I tried running

python main.py --data_dir input_images --latent_path latents.pt --protected_face_dir results

I got

AttributeError: 'dict' object has no attribute 'eval'

This is caused in line 209 in adversarial_optimization.py.
i.e g_ema = torch.load(self.generators[ff]).eval() #loading fine-tuned generator

Generally, we use model.load(weight.pt) after creating the model instance model=Model(). Is the above error because of this or this is arising because of different Pytorch versions?

Paired Data for Dodging Attack

Thanks for your excellent work! Could you provide paired data for CelebA-HQ and LFW for face verification and face identification tasks?

Could you provide the selected 1,000 from CelebA-HQ and 4 target identities used in the experiments of the paper?

Thanks for open-sourcing the wonderful work.
We would like to build our own work based on Clip2Protect.

The paper writes:

We select subset of 1,000 images from CelebA-HQ and report average results over 4 target identities provided by [22].

Could you provide the selected 1,000 from CelebA-HQ and 4 target identities used in the experiments of the paper?

Thanks a lot.

Maybe a bug in your code

I found a bug in your code.
In pivot_tuning.py, line 79, it is supposed to be self.noise_save.append(noise_), instead of self.noise_save.append(noises), which leads to continuously increasing cuda usage.

Some questions about the lfw experiment

Sorry to bother you, looked at your public code and it is very well written. Following your process, I selected 6 pieces of data from the lfw dataset and obtained their own latent code as well as the stitched together implicit code (obtaining the latent code for each requires modifying e4e's code), which I then put into your code for finetune and generation, but the encrypted image that ended up being generated with the red lips is very strange, and has a lot of and the original image has a big difference, and the first phase of the inverse image generated is not very clear, can I ask you where this problem is?

LADN split

Hi,

Thanks for great work,

Do you have the split file of LADN dataset into 4 groups and the target Id. I try to look at the AMT-GAN paper and its github repository, but I am still unable to see it.

Thank you,

How can we use your method when the user do not want to change anything attribute on his face

hello! Your paper is very interesting. But I have some question about application scenario. When user do not want to change anything attribute about his face, how your method can work to protect privacy. I very hope that I can get chance to communicate with you.

Add the ONNX version

Add the ONNX version

Some questions about the calculation of protection success rate(PSR) and dodging attack.

Your work is excellent, but I have a few questions:

1. In the adversarial loss for dodging attacks, why is the cosine distance between the generated identity and the target identity used as the first term of the adversarial loss? As far as I know, dodging attacks should not require a target identity to guide, that is, it is not necessary to minimize the cosine distance between the generated identity and the target identity (the first term of the adversarial loss). It is only necessary for the distance between the generated identity and the original identity to be large enough (the second term of the adversarial loss).
2. When calculating the protection success rate(PSR), you calculate the cosine similarity between the generated portrait (i.e., the protected portrait) and the target portrait identity in the function "black_box". Then, in the function "quan", you consider the cosine similarity is greater than the system threshold τ as a successful attack. My understanding is that the cosine similarity should be greater than (1-τ) , this is a successful protection, or the cosine distance should be less than the τ.
3. If the first point is correct, that is, dodging attacks involves two the adversarial losses, then in order to ensure that the protected image can be identified as the target identity, the cosine distance between the protected image and the target identity should be less than the cosine distance between the protected image and the original identity. That is, the optimization should be terminated when the adversarial loss is less than 0. However, you terminated the optimization after only 50 iterations, and I think it's too early to terminate and the protection effect seems not very good.
   I would be very grateful if you could reply to me.