jtchen0528 / pcl-i2g

Unofficial Implementation: Learning Self-Consistency for Deepfake Detection

PCL-I2G

Unofficial implementation of paper: Learning Self-Consistency for Deepfake Detection (ICCV2021)

Disclaimer

This repo is still developing (training). Please open an issue or contact me if you have any questions.

• Forked/Modified from chail/patch-forensics (Thanks!)

Installation

1. create conda environment with Python=3.7

   conda create -n PCL-I2G python=3.7
   conda activate PCL-I2G

2. install pytorch 1.9.0, torchvision 0.10.0 with compatible cuda version

   pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

3. install required packages

   pip install -r requirements.txt

Dataset

Basically any real data works on the methodology, but here I use FaceForensics++.

• FaceForensics++

Pre-processing

1. extract frames from videos (with ffmpeg)
2. Download dlib landmark predictor

   # dlib facial landmarks predictor
   wget -O resources/shape_predictor_68_face_landmarks.dat.bz2 https://github.com/davisking/dlib-models/raw/master/shape_predictor_68_face_landmarks.dat.bz2
   bzip2 -d resources/shape_predictor_68_face_landmarks.dat.bz2

3. run data/preprocessing/faceforensics_process_frames.py

   python -m data.processing.faceforensics_process_frames.py \
       --source_dir_manipulated $manipulated \
       --source_dir_original $original \
       --outsize $outsize \
       --output_dir $out\
       --split resources/splits/train.json
       # --split resources/splits/val.json
       # --split resources/splits/test.json

   or run scripts/preprocess.sh

   bash scripts/train.sh

Inconsistency Image Generator (I2G)

1. run generate_I2G.py

   python generate_I2G.py \
       --real_im_path $original \
       --batch_size 512 \
       --out_size 256 \
       --output_dir $out

Pair-Wise Self-Consistency Learning (PCL)

Training

run train.py:

python train.py \
	--gpu_ids $gpu --seed 0 --loadSize 256 --fineSize 256 \
	--name PCL-I2G-FF256-32frames-Modified-5e-5 --save_epoch_freq 10 \
 	--real_im_path $dset/real/face \
 	--fake_im_path $dset/fake/face \
	--which_model_netD resnet34_layer4_extra3 \
    --model patch_inconsistency_discriminator \
	--lbda 10 --patience 5 --lr_policy constant \
    --max_epochs 500 --batch_size 512 --lr 5e-5 \
	--overwrite_config

or run train_PCL.sh

bash scrips/train_PCL.sh

Argument --which_model_netD resnet34_layer4_extra3 and --model patch_inconsistency_discriminator is the model architecture used in the paper.

Testing

run test.py

python test.py --which_epoch $which_epoch --gpu_ids $gpu --partition $partition \
    --average_mode after_softmax --topn 100 --force_redo \
    --dataset_name FF-DF \
    --real_im_path $dset/original/$partition \
    --fake_im_path $dset/DF/$partition \
    --train_config $checkpoint/opt.yml

Learned result on 4D consistency volume

pcl-i2g's Issues

About masks when learning with FF++

Hello, thanks for your code implementation.
Please tell me how to prepare the mask when training on the FF++ dataset?

About Pre-process

100%|███████████████████████████████████████████████████████████████████████████████| 70/70 [00:00<00:00, 69955.03it/s]

after this ，my out file has only empty folder ，no image or mask

Can you provide the pretrained model on FF++ (HQ)?

What's the function of celebahq_crop when preprocess frames?

Hello, could you explain what's the function of celebahq_crop when preprocess frames is? I'm not very clear about that.
Thank U!

Are the results the same as those in the original paper?

Hello, thanks for your code implementation. I believe the results of in-domain (train and test data from the same dataset) in the original paper are correct, while the out-of-domain (train and test data from different datasets) results are too suprising to me. Previous works, which uses temporal information, can achieve about 0.7-0.8 AUC in CDF (up to ICCV 2021). I really hope that you can answer my question after some experiment.

By the way, few researchers can afford the raw version of FF++ videos : ).

Low AUC in FF-DF

Hi! I re-train a new PCL model by following your codes. The training data is only I2G data. However, the AUC of FF-DF is only 0.5. It seems like that the trained model has no ability of generalization. I have checked the log file, which shows that loss_D is 1.1+ and acc_D is 1.0 in the training phase, but loss_D and acc_D in the validation phase are 17+ and 0.5, respectively. I have no idea about the reason of that. Would you like to give me some suggestions about this problem? Maybe the generated I2G data or some possible errors? And would you like to provide your trained weight of PCL? My email is [email protected]. Thanks you a lot~

I2G DATA

Hi, can you share I2G dataset from FF++, because I meet some errors, but I only want the dataset to train my model, thanks. email is [email protected]

How can I find the original real images？

Excuse me，could you please tell me how I can find the original real images？how can I use the FaceForensics++？And I don't know why I need extract frames from videos (with ffmpeg) at the first step of preprocessing. Thank you!

Mask size error

Thanks for your work! I'm trying to conduct experiments on the settings of
----------------- Options ---------------
batch_size: 32
beta1: 0.9
checkpoints_dir: ./checkpoints
display_freq: 1000
fake_class_id: 0
fake_im_path: ./dataset\DF
fineSize: 224
gpu_ids: [0] [default: 0]
help: None [default: ==SUPPRESS==]
init_type: xavier
isTrain: True [default: None]
lbda: 10
loadSize: 256
load_model: False
lr: 0.001
lr_policy: constant
max_dataset_size: inf
max_epochs: None
model: patch_inconsistency_discriminator
nThreads: 0
name: patch_inconsistency_discriminator_resnet34_layer4_extra3_size224 [default: ]
overwrite_config: True [default: False]
patience: 10
prefix:
print_freq: 100
real_im_path: ./dataset\original
results_dir: ./results/
save_epoch_freq: 100
save_latest_freq: 1000
seed: 0
suffix:
which_epoch: latest
which_model_netD: resnet34_layer4_extra3
----------------- End -------------------
However, during the training process, i run into this issuse:
File "D:\pythonProject\PCL-I2G\models\patch_inconsistency_discriminator_model.py", line 85, in compute_losses_D
masks = self.mask_down_sampling(masks).reshape(n, h, w)
RuntimeError: shape '[32, 14, 14]' is invalid for input of size 8192

Could you please give me some advice on how to solve this? Thank！

Missing code for generating FF++dataset mask

I trained a model using the I2G dataset, but the AUC was very low. The paper states that it is necessary to mix the FF++dataset with the I2G dataset for training.
The paper states that”The first type is the deepfake video frame from the existing deepfake datasets, for which we find the corresponding real video frame and compute the structural dissimilarity (DSSIM) [60] between them. The mask M is then generated by taking a Gaussian blur of DSSIM followed by thresholding.“
However, it seems that this part of the code is not provided in the paper, resulting in the FF+dataset not having a mask. Can you provide guidance.

Would you give me an advice of 'masks' of input images?

I have trained a model, named 'PCL-I2G-FF256-32frames-Modified-5e-5'. When I run the test.py with this setting below:

/public/user/anaconda3/envs/PCL-I2G/bin/python3.7 test.py --which_epoch bestval --gpu_ids 0 --partition test --average_mode after_softmax --topn 100 --force_redo --dataset_name FF-DF --real_im_path /data-x/g16/user/outputdir/original/test/ --fake_im_path /data-x/g16/user/outputdir/F2F/test/ --train_config /public/user/PCL-I2G/checkpoints_back/PCL-I2G-FF256-32frames-Modified-5e-5/opt.yml --checkpoints_dir /public/user/PCL-I2G/checkpoints_back/

an error is reported. It looks like I need a mask as an input for testing images.

Traceback (most recent call last):
  File "test.py", line 257, in <module>
    run_eval(opt, output_dir)
  File "test.py", line 62, in run_eval
    model.set_input(inputs)
  File "/public/user/PCL-I2G/models/patch_inconsistency_discriminator_model.py", line 66, in set_input
    self.masks = input['masks'].to(self.device)
KeyError: 'masks'

Would you tell me how to get the mask of testing images? Thanks.

Generation of I2G data error

Excuse me, when I try to generate I2G data with
python3 generate_I2G.py --real_im_pa dataset --batch_size 512 --out_size 256 --output_dir out
with FF++ dataset, the following error occurs The following error occurs.
The data used is a frame of a real image extracted from the FF++ dataset.
Do you know what the cause is?

-----error-----
ValueError: Sample larger than population or is negative

Could you please give me some advice on how to solve this? Thank！

About Pre-processing

Dear ,
I want to ask about Pre-processing part
the second step:
Download dlib landmark predictor
How to use this