sutd-visual-computing-group / re-thinking_mi

[CVPR-2023] Re-thinking Model Inversion Attacks Against Deep Neural Networks

Implementation of paper "Re-thinking Model Inversion Attacks Against Deep Neural Networks" - CVPR 2023

Paper | Project page

1. Setup Environment

This code has been tested with Python 3.7, PyTorch 1.11.0 and Cuda 11.3.

conda create -n MI python=3.7

conda activate MI

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

pip install -r requirements.txt

2. Prepare Dataset & Checkpoints

• Dowload CelebA and FFHQ dataset at the official website.

• CelebA: download and extract the CelebA. Then, place the img_align_celeba folder to .\datasets\celeba

• FFHQ: download and extract the FFHQ. Then, place the thumbnails128x128 folder to .\datasets\ffhq

• Download meta data for the experiments at: https://drive.google.com/drive/folders/1kq4ArFiPmCWYKY7iiV0WxxUSXtP70bFQ?usp=sharing

• We use the same target models and GAN as previous papers. You can download target models and generator at https://drive.google.com/drive/folders/1kq4ArFiPmCWYKY7iiV0WxxUSXtP70bFQ?usp=sharing

Otherwise, you can train the target classifier and GAN as follow:

2.1. Training the target classifier (Optional)

• Modify the configuration in .\config\celeba\classify.json
• Then, run the following command line to get the target model

  python train_classifier.py
  

2.2. Training GAN (Optional)

SOTA MI attacks work with a general GAN[1]. However, Inversion-Specific GANs[2] help improve the attack accuracy. In this repo, we provide codes for both training general GAN and Inversion-Specific GAN.

2.2.1. Build a inversion-specific GAN

• Modify the configuration in

  • ./config/celeba/training_GAN/specific_gan/celeba.json if training a Inversion-Specific GAN on CelebA (KEDMI[2]).
  • ./config/celeba/training_GAN/specific_gan/ffhq.json if training a Inversion-Specific GAN on FFHQ (KEDMI[2]).

• Then, run the following command line to get the Inversion-Specific GAN

  python train_gan.py --configs path/to/config.json --mode "specific"
  

2.2.2. Build a general GAN

• Modify the configuration in

  • ./config/celeba/training_GAN/general_gan/celeba.json if training a general GAN on CelebA (GMI[1]).
  • ./config/celeba/training_GAN/general_gan/ffhq.json if training a general GAN on FFHQ (GMI[1]).

• Then, run the following command line to get the General GAN

  python train_gan.py --configs path/to/config.json --mode "general"
  

3. Learn augmented models

We provide code to train augmented models (i.e., efficientnet_b0, efficientnet_b1, and efficientnet_b2) from a target model.

• Modify the configuration in

  • ./config/celeba/training_augmodel/celeba.json if training an augmented model on CelebA
  • ./config/celeba/training_augmodel/ffhq.json if training an augmented model on FFHQ

• Then, run the following command line to train augmented models

  python train_augmented_model.py --configs path/to/config.json
  

Pretrained augmented models and p_reg can be downloaded at https://drive.google.com/drive/u/2/folders/1kq4ArFiPmCWYKY7iiV0WxxUSXtP70bFQ

We remark that if you train augmented models, please do not use our p_reg. Delete files in ./p_reg/ before inversion. Our code will automatically estimate p_reg with new augmented models.

4. Model Inversion Attack

• Modify the configuration in

  • ./config/celeba/attacking/celeba.json if training an augmented model on CelebA
  • ./config/celeba/attacking/ffhq.json if training an augmented model on FFHQ

• Important arguments:

  • method: select the method either gmi or kedmi
  • variant select the variant either baseline, L_aug, L_logit, or ours

• Then, run the following command line to attack

  python recovery.py --configs path/to/config.json
  

5. Evaluation

After attack, use the same configuration file to run the following command line to get the result:\

python evaluation.py --configs path/to/config.json

Acknowledgements

We gratefully acknowledge the following works:

• Knowledge-Enriched-Distributional-Model-Inversion-Attacks: https://github.com/SCccc21/Knowledge-Enriched-DMI
• EfficientNet (Pytorch): https://pytorch.org/vision/stable/models/efficientnet.html
• Experiment Tracking with Weights and Biases : https://www.wandb.com/

Reference

[1] Zhang, Yuheng, et al. "The secret revealer: Generative model-inversion attacks against deep neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[2] Si Chen, Mostafa Kahla, Ruoxi Jia, and Guo-Jun Qi. Knowledge-enriched distributional model inversion attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16178–16187, 2021

re-thinking_mi's Issues

The acc of the provided checkpoint is very low

Hello!
I used the checkpoints of the target classifier VGG16 and IR152 provided by you and tested the classification accuracy.
Why is the acc only 9.97%?

KeyError: 'gan_file_path'

Hi,

I am currently trying to reproduce the results from the paper. However, I am not able to run the LOMMA attack. However, it seems that a key (gan_file_path) is missing in the configuration file:

Traceback (most recent call last):  
  File "recovery.py", line 75, in <module>
    targetnets, E, G, D, n_classes, fea_mean, fea_logvar = get_attack_model(  
  File "/workspace/utils.py", line 289, in get_attack_model  
    _, dataloader_gan = init_dataloader(args_json, args_json['dataset']['gan_file_path'], 50, mode="gan")  
KeyError: 'gan_file_path'  

And this is my configuration file:

{
    "root_path": "./attack_results_lomla_00/",
    "dataset": {
        "model_name": "VGG16",
        "test_file_path": "./datasets/celeba/meta/testset.txt",
        "name": "celeba",
        "img_path": "./datasets/celeba/img_align_celeba",
        "n_classes": 1000,
        "fid_real_path": "./datasets/celeba/meta/celeba_target_300ids.npy",
        "KNN_real_path": "./datasets/celeba/meta/fea_target_300ids.npy",
        "p_reg_path": "./checkpoints/p_reg",
    },
    "train": {
        "model_types": "VGG16,efficientnet_b0",
        "cls_ckpts": "checkpoints/target_model/target_ckp/VGG16_85.74_allclass.tar,checkpoints/aug_ckp/celeba/VGG16_efficientnet_b0_0.01_1.0/VGG16_efficientnet_b0_kd_1_20.pt",
        "num_seeds": 5,
        "Nclass": 1000,
        "gan_model_dir": "./checkpoints/GAN",
        "eval_model": "FaceNet",
        "eval_dir": "./checkpoints/target_model/target_ckp/FaceNet_95.88.tar"
    },
    "attack": {
        "method": "gmi",
        "variant": "ours",
        "iters_mi": 2400,
        "lr": 0.02,
        "lam": 1.0,
        "same_z": "",
        "eval_metric": "fid, acc, knn"
    }
}

Could you help me to fix this bug? I already tried adding the missing key with the value set to the generator path. Still, I receive another error for this. Thank you very much in advance :)

Best,
Lukas

Training target model

Sorry.