ahaliassos / lipforensics Goto Github PK

Can you provide your train.py file?

I use python version and requirements.txt specified in this repo to create a python env. For videos from val and test splits only first 110 frames are used for evaluation. I enlarge the face bounding boxes found by RetinaFace by 1.3 times preserving their centers and feed them into FAN to get landmarks. Only one bbox per face is selected - the one with highest model confidence. Then I use your scripts for mouth cropping and AUC calculation for published ckeckpoint "lipforensics_ff.pth".

Video-level AUCs which I get on test split of FaceForensics++ with different compression levels are lower than the scores in Table 4 of your paper:
Raw-99.9, HQ-99.7, LQ-98.1 against my results obtained for "lipforensics_ff.pth": Raw-98.9, HQ-98.7, LQ-81.3.

So the first question is - did you use FF++ train set with only one compression level to get published ckeckpoint "lipforensics_ff.pth"?

If the answer is yes and only one compression level was present in the train set, do you have any ideas why the score I get on the FF++ test set with this compression level is at least 1% lower than yours?

Hi,

Is it possible to provide the 4 models used for the generalisation to unseen manipulations testing used to produce table 1 in the paper?

Please provide the training code.
Thanks!

I follow your instruction to prepare CelebDF test dataset, trying to validate lipforensics_ff.pth on CelebDF. However I came across:

Traceback (most recent call last):
  File "evaluate.py", line 179, in <module>
    main()
  File "evaluate.py", line 174, in main
    auc = validate_video_level(model, loader, args)
  File "evaluate.py", line 109, in validate_video_level
    for data in tqdm(loader):
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/users/kaixuanzhang/repos/LipForensics/data/dataset_clips.py", line 169, in __getitem__
    sample = self.transform(sample)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 70, in __call__
    img = t(img)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 247, in __call__
    return F.center_crop(img, self.size)
  File "/data/users/kaixuanzhang/anaconda3/envs/torch14/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 382, in center_crop
    image_width, image_height = img.size
TypeError: 'builtin_function_or_method' object is not iterable

I noticed that the transform input is np.ndarray but the PIL Image is required, so I substitute the CenterCrop with VideoCenterCrop:

class VideoCenterCrop:
    def __init__(self, outputsize):
        self.output_size = outputsize

    def __call__(self, img):
        """
        param img (numpy.ndarray): THWC 
        """
        if isinstance(self.output_size, numbers.Number):
            output_size = (int(self.output_size), int(self.output_size))
        #print("img shape", img.shape)
        image_width, image_height = img.shape[2], img.shape[1]
        crop_height, crop_width = self.output_size
        crop_top = int(round((image_height - crop_height) / 2.))
        crop_left = int(round((image_width - crop_width) / 2.))
        return img[:, crop_top:crop_top + crop_height, crop_left:crop_left + crop_width, :]

Then run

python evaluate.py --dataset CelebDF --weights_forgery ./models/weights/lipforensics_ff.pth

It works, but I got only AUC of 57.7% on CelebDF. I am wondering which step did I get wrong.
I really appreciate it if you could give me some advice. Thank you!

Hi,thank you for sharing and I want to ask for some help

my scipy version is 1.5.4 and I cant install the 1.7.1

when I run the code ,it make a mistake

Processing Face2Face...
0%| | 0/1000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "preprocessing/crop_mouths.py", line 199, in
main()
File "preprocessing/crop_mouths.py", line 195, in main
crop_video_and_save(video_dir, landmarks_dir, target_dir, mean_face_landmarks, args)
File "preprocessing/crop_mouths.py", line 122, in crop_video_and_save
smoothed_landmarks[STABLE_POINTS, :], mean_face_landmarks[STABLE_POINTS, :], cur_frame, STD_SIZE
File "F:\master-data\deepfake\LipForensics\preprocessing\utils.py", line 24, in warp_img
tform = tf.estimate_transform("similarity", src, dst) # find the transformation matrix
File "F:\Anaconda3\envs\pytorch19\lib\site-packages\skimage\transform_geometric.py", line 1408, in estimate_transform
tform.estimate(src, dst, **kwargs)
File "F:\Anaconda3\envs\pytorch19\lib\site-packages\skimage\transform_geometric.py", line 1169, in estimate
self.params = _umeyama(src, dst, True)
File "F:\Anaconda3\envs\pytorch19\lib\site-packages\skimage\transform_geometric.py", line 109, in _umeyama
A = dst_demean.T @ src_demean / num
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 960 is different from 5)

To predict as real or fake from logits, can I sigmoid(logits) and put some threshold say 0.3 to classify as real or fake ? Or shall I directly threshold logits ? If yes can you suggest some threshold ? Thanks

Hello, because I want to use your method to conduct a compare experiment, I feel that your method is effective and interesting, so I want to try it. Are you convenient to provide your train.py file?

According to your picture, the final layer of the model should decide whether the input is true or fake (0/1), but the model spatiotemporal_net.py does not have a softmax layer at the end. Instead, the output of the model is a number that ranges across -50~50 (or some other number that isn't between 0 and 1).
Does this mean that there is another way to calculate the loss of a classifier when training?
If there indeed is another loss function besides binary cross entropy, can you share it with us?

How does the output logits value judge whether it is real or fake?