dfaker / df

Larger resolution face masked, weirdly warped, deepfake,

df

Larger resolution face masked, weirdly warped, deepfake, requires https://github.com/keras-team/keras-contrib.git and https://github.com/1adrianb/face-alignment.git to be cloned into the root of the repo and A and B's alignments.json to be copied into the correct /data/* folder before training.

Inputs are 64x64 images outputs are a pair of 128x128 images one RGB with the reconstructed face, one B/W to be used as a mask to guide what sections of the image are to be replaced.

For the reconstrcuted face masked DSSIM loss is used that behaves as a standard SSIM difference measure in the central face area and always returns zero loss in the surrounding background area outside of the face so as as not to train irrelevant features.

MSE is used for the mask.

Training

Merged Face Extracts

Final output

Guide

• Clone this repository
• cd into the folder df and clone https://github.com/keras-team/keras-contrib.git
• make sure that the folder keras_contrib is in the root of the df respository
• cd into the folder df and clone https://github.com/1adrianb/face-alignment.git
• make sure that the folder face-alignment is in the root of the df respository
• run align_images_masked on your source A and B image folders.
• copy the aligned cropped images into the A or B folder along with the alignments.json files from the source image folders.
• run train.py as usual
• wait
• run merge_faces_larger.py on your image folder.

Directory structure for training data should look like (image names for example purposes only):

df
│
└───data
    │
    ├───A
    │   │ alignments.json
    │   │ trainingimageA1.jpg
    │   │ trainingimageA2.jpg
    │
    ├───B
    │   │ alignments.json
    │   │ trainingimageB1.jpg
    │   │ trainingimageB2.jpg

or as a file listing:

.\df\data\A\alignments.json
.\df\data\A\trainingimageA1.jpg
.\df\data\A\trainingimageA2.jpg
.\df\data\B\alignments.json
.\df\data\B\trainingimageB1.jpg
.\df\data\B\trainingimageB2.jpg

imageGapsScanner.py

Image gaps scanner can be run with this command to scan both the A and B folders:

imageGapsScanner.py data\A data\B

it will pop open a new window showing two rows, the images in A that have the worst matches in B and the images in B that have the worst matches in A: In that example the top row has some clear images that are totally different to the data in B, ideally these would be hunted down, you'd try to find some images of cage shouting at various angles, of the examples in the bottom row however only the first seems to be of a concern, you would consider deleting that image or finding a matching trump image for it, for the remaining images in the bottom row, although they are the top 9 "worst" matches they seem to be relatively nomal front-on shots and can be ignored.

df's Issues

align_images_masked.py assert input_dir.is_dir()

I'm trying to prepare images for train, but getting error with align_images_masked.py
python3 align_images_masked.py input_dir=/home/tuw/Desktop/df/dp2

Traceback (most recent call last):
  File "align_images_masked.py", line 125, in <module>
    main( parser.parse_args() )
  File "align_images_masked.py", line 53, in main
    assert input_dir.is_dir()
AssertionError

When i change line #112 from parser.add_argument( "input_dir" , type=str )
to
parser.add_argument( "input_dir" , type=str, nargs='?', default='/home/tuw/Desktop/df/dp2' )It worked
How to fix it? Edit every time source code is for input dir is 😩

Need for alignments.json during training

If you attempt to run train.py without having alignments.json files on both A and B folders, the training process will not be allowed to execute. I know that other implementations of this software, such as FakeApp, don't require this file for training, but I am not sure if it is necessary for this particular implementation. If it is not, it would be great if this requirement could be removed on a future version since adding more images for training would, to my knowledge, require you to run the whole extraction process again on all images.

ImportError: DLL load failed: %1 is not a valid Win32 application.

When im trying to run align_images_masked.py
im getting
Traceback (most recent call last):
File "align_images_masked.py", line 2, in
import cv2
File "C:\Users\Admin\Anaconda3\lib\site-packages\cv2_init_.py", line 4, in
from .cv2 import *
ImportError: DLL load failed: %1 is not a valid Win32 application.
What the problem?

ImportError: No module named face_alignment

First question, does df need python2 or python3?
And when i'm trying to run python align_images_masked.py or python3 align_images_masked.py
I'm getting

python3 align_images_masked.py

Traceback (most recent call last):
  File "align_images_masked.py", line 11, in <module>
    from face_alignment import FaceAlignment, LandmarksType
ImportError: No module named face_alignment

Tried to install face_alignment module, but
pip3 install face_alignment

Collecting face_alignment
  Could not find a version that satisfies the requirement face_alignment (from versions: )
No matching distribution found for face_alignment
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

How i can install face_alignment module? p.s. im on Ubuntu 16.04

align_images_masked.py is not using GPU...

It's normal or should be using the GPU?

ModuleNotFoundError: No module named 'face_alignment' (Last Repo).

FIXED.

Enable multiple gpus

Tried to enable processing ok two gpus, but got an error about layers having the same name. Any easy way to get this working?

Kernel and Blur size, doesn't seems to change...

I changed the values inside to merge_faces_larger.py

parser.add_argument( "--blurSize",          type=int, default='250' )
parser.add_argument( "--erosionKernelSize", type=int, default='170' )

(Like do with the APP 1.1)

But it doesn't make any change, how i can reduce the surface size of the "fake" face i'm adding over the original face?

KeyError: 'unexpected key "state_dict" in state_dict'

Tried to find a solution but i couldn't get one...

Need to change something there or...?

face_aligment depencies

Hi, in align_images_masked.py

Where or how to install this package?

IndexError: list index out of range

(dfaker) D:\Library\Desktop\DFaker>python train.py
running
Traceback (most recent call last):
  File "train.py", line 25, in <module>
    images_A,landmarks_A = load_images( images_A[:minImages] )
  File "D:\Library\Desktop\DFaker\utils.py", line 14, in load_images_masked
    basePath = os.path.split(image_paths[0])[0]
IndexError: list index out of range

(dfaker) D:\Library\Desktop\DFaker>

Adding this repo as deepfakes/faceswap plugin

Hi @dfaker ,

Just opening this so we can discuss about adding your work as a plugin in the deepfakes/faceswap repo. I don't know if you had this in mind but I started it on my side but did not went very far (https://github.com/Clorr/df). For now I stopped, but if you are interested in helping me to make this happen, let's discuss it here ;-)

How much size supported most?

Thanks for this work, as far as I know, the original deepfakes only support 64x64 as input, which is determined by Linear layer of model ( can not change input size ). How did this repo maxium size support? 256 or ?

What do I write in the command line with align_images_masked.py?

Merging process

if i want to do merging process, what i should i type in cmd?? is this the way??

python merge_faces_larger.py -i ~df/data/A/ -o ~df/output/ -m ~df/models/ --seamlessClone True --blurSize 10 --erosionKernelSize 7 .

Why not last TENSORFLOW in requirements.txt?

tensorflow-gpu==1.4.0 in requirements.txt

It's not compatible?

Need for facial alignment

I wanted to ask if it is necessary to do the face_alignment before training? Why is it necessary? What will happen if I train the model without aligned images?

Consider using face_segmentation library

Consider using https://github.com/YuvalNirkin/face_segmentation library for better area coverage of the face #

Images in input folder not being detected

I am running align_images_masked.py but when I enter "python align_images_masked.py sourceA" I get the message below:

(df) C:\df>python align_images_masked.py sourceA aligned
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xa
Traceback (most recent call last):
File "align_images_masked.py", line 125, in
main( parser.parse_args() )
File "align_images_masked.py", line 68, in main
assert len( input_files ) < 0, "Can't find input files"
AssertionError: Can't find input files

sourceA is where my images are stored and there are 100 of them. I've tried converting them to png, jpg, and bmp. Every time the script says it can't find the input files, but it does create an empty aligned folder in sourceA, so I know it's partly working. I am using windows 10 x64 and anaconda. My files are setup like this:

C:/df/sourceA <------100 images in here

The script is located here: C:/df/align_images_masked.py

merge_faces_larger.py TypeError: unhashable type: 'slice'

Hi,

I would like to for help with the following problem:
merge_faces_larger.py says:

2018-10-04 19:23:05.077752: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Traceback (most recent call last):
File "merge_faces_larger.py", line 244, in
main( parser.parse_args() )
File "merge_faces_larger.py", line 202, in main
for image_file, face_file, mat,facepoints in tqdm( alignments[args.startframe::args.frameSkip] ):
TypeError: unhashable type: 'slice'

tqdm is 4.26.0

Could you please help me what could be the problem?
Thank you!
Br,Herold

Alignment with Cuda fails

Any obvious reason why alignment with Cuda enabled fails on larger resolution images? The model itself gives incredible results, so its so frustrating that the alignment tool doesnt work with GPU enabled.

I tried using the updated alignment tool from the deepfake/faceswap repo but I guess the .json output is different.

Maybe it could be rewritten to fit the format dfaker needs? Alignment on 56 core CPU (Dual xenon) takes forever compared to GPU

How can I change layers and nodes

What is the default number of layer and nodes and how can I change them or other options.

MemoryError

Images A (649, 256, 256, 4) Images B (1041, 256, 256, 4) Traceback (most recent call last): File "train.py", line 31, in images_A = images_A/255.0 MemoryError

What could be the problem here? Not enough RAM? I have 11GB free at the point of running the script.

Post some results enchancement

Hello, does this currently work well on hight resolution face swap?

Models breaking after several hours

It's happened many times now. After a few hours the preview will suddenly change from attempting to merge the face to a changing blob of solid colors. After a while the colors seem to settle on red and from then on the model becomes broken and, as far as I know, unfixable unless you have a backup. Sometimes it's the third row of each image which goes completely red and sometimes it's the second, regardless attempting the merge process while the models are on this broken red state results in blank faces in the final output.

the converted image is the same as the previous one.

after train the model ,i get the 'encoder.h5' 'decoder_A.h5' 'decoder_B.h5'
when i use these file to run 'merge_face_larger.py'
the input dir is data/B:
├─merged
│ │ alignments.json
│ │ trainingimageB1.jpg
│ │ trainingimageB2.jpg

output dir is data/B/merged.
But the converted image is the same as the previous one.

Empty 'merged' folder?

Hello,
I trained a model and it all looks very good from the preview window.

Now, I'm trying to create the merged faces, and when I run merge_images_larger.py on the original images folder (not the faceset), I do get a progress bar, and a "merged" folder is created. But it is empty.

I tried adding parameters etc. but to no avail.. always same thing. Any thoughts of anything I could try to troubleshoot this?

Out of VRAM

While running train.py I'm running out of GPU memory. I already tried to set the batch size down to 4 without any improvement. Can you recommend any model parameters to adapt? My GPU is a GTX 970 4GB (of which Tensorflow can only use 3.5GB).

The error is: OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[4,16,16,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Allocation Stats:

Stats: 
Limit:                  3432906752
InUse:                  3428587264
MaxInUse:               3431733248
NumAllocs:                    1546
MaxAllocSize:            597213184

EDIT: I tried to set ENCODER_DIM to 256 and I don't get Memory errors anymore, but now I'm presented with this error:

Traceback (most recent call last):
  File "train.py", line 151, in <module>
    figure = figure.reshape( (4,4) + figure.shape[1:] )
ValueError: cannot reshape array of size 589824 into shape (4,4,3,128,128,3)

There is no “load_images ” in utils

Monkey patch in face detector

Any particular reason replacing cnn face detection with frontal face detection?

What will be script if run at cloud base gpu floyhub

I don't have strong GPU, but think to try it at cloud gpu floyhub. What will be scrip to run it ?

Installing face-alignment in Windows

How are others getting the face-alignment dependency to install under Windows 10? I've tried anaconda/miniconda, and the torch dependency fails during setup. I haven't been able to successfully build torch in Windows myself or get a working install independently. From looking at other people's screenshots, it looks like they are not bothering to install face-alignment?

Algorithm explanation

Hi dfaker, if you don't mind, can I ask some questions about the implementation? I'm not sure where else to ask other than here.

For the larger faces, I see that you pull out 240x240 pixels from the 256x256 cropped faces. The original code uses 160x160, like at the faceswap repo.

Is anything else in your code specific to generating larger face models? The model is more complex, but I think that most of it is related to generating a higher resolution 128x128 output, not for generating larger face models?

In other words, if I took the faceswap repo, and just change the image_augmentation to mimic yours with 240x240, would that generate large face models at 64x64 resolution? Or is there some other step that needs to be changed to make it work? Are changes in the conversion process essential to the large face model, or wouldn't just using user-defined kernel/blur etc. be sufficient?

[del]

What kind of VRAM requirements are necessary for this fork of deepfakes

I ask because, unlike in deepfakes/faceswap or FakeApp, my GPU seems to not have enough VRAM for the training process, spitting out the following error (uploaded to hastebin due to its sheer size): https://hastebin.com/raw/metojewuqa

It is a GTX 1060 6GB for reference, with 6GB of VRAM (as I'm sure you could tell). Is this not enough for this project? Has the model been increased to 128x128 perhaps? If so, is there a way to bump it back down to 64x64 or perhaps a middle-ground like 96x96.

Thank you.

Why not utilize blur/kernel size

I'm just curious why you don't utilize blur/kernel size settings during the merging process. Opencv's seamlessClone can produce decent results, but in my experience it leads to flickering faces more often than not and isn't concealing the boundary of the output face as effectively as the right blur/kernel size settings.

how to apply a swap on a particular part

Hi,
I'm interested in applying a swap on a particular part of a face.
e.g eyes or mouth.
what would be the best approach for that?

Thanks!

While training I sometimes get this crash

And training stops.
Don't have this problem with original scripts.

[02:46<563:58:07, 2.03s/it]2018-02-06 19:49:27.301416: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 128.00MiB. Current allocation summary follows.
2018-02-06 19:49:27.301575: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:627] Bin (256): Total Chunks: 96, Chunks in use: 93. 24.0KiB allocated for chunks. 23.3KiB in use in bin. 484B client-requested in use in bin.
2018-02-06 19:49:27.301714: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:627] Bin (512): Total Chunks: 11, Chunks in use: 10. 5.5KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin.
2018-02-06 19:49:27.301983: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:627] Bin (1024): Total Chunks: 27, Chunks in use: 27. 27.3KiB allocated for chunks. 27.3KiB in use in bin. 27.0KiB client-requested in use in bin.

...

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[32,64,128,128]
[[Node: training/Adam/gradients/model_2/conv2d_16/convolution_grad/Conv2DBackpropInput = Conv2DBackpropInput[T=DT_FLOAT, _class=["loc:@model_2/conv2d_16/convolution"], data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/model_2/conv2d_16/convolution_grad/ShapeN, conv2d_16/kernel/read, training/Adam/gradients/model_2/conv2d_16/Sigmoid_grad/SigmoidGrad)]]

IndexError: too many indices for array

Microsoft Windows [Versión 10.0.16299.98]
(c) 2017 Microsoft Corporation. Todos los derechos reservados.

C:\WINDOWS\system32>cd C:\df-master

C:\df-master>python train.py
running
loading data/A: 100%|███████████████████████████████████████████████████████████████| 190/190 [00:00<00:00, 445.43it/s]
loading data/B: 100%|█████████████████████████████████████████████████████████████| 946/946 [00:00<00:00, 18816.39it/s]
Images A (125, 256, 256, 4)
Images B (0,)
Traceback (most recent call last):
File "train.py", line 34, in
images_A[:,:,:3] += images_B[:,:,:3].mean( axis=(0,1,2) ) - images_A[:,:,:3].mean( axis=(0,1,2) )
IndexError: too many indices for array

C:\df-master>

My model is fully trained but now I can't swap out the faces

I'll pretend I'm swapping Trump's State of the Union vid with Nic Cage's face so we don't get confused.

I run 'run merge_faces_larger.py',
that is I type 'python merge_face_larger.py' 'path to photos ripped from State of the Union Trump video'

The terminal starts chunking through the photos, a screen pops up that shows each frame but faces are not being swapped.
When the process is done I have a file of photos called 'merged' and the photos are the exact same photos it started with. Nothing swapped.
Picture me with a Nic Cage 'A', fully trained Trump 'B' model, run on the Trump video frames and only returned Trump video frames in the 'merged' folder - just to be clear about it.

Something is not right... (VIDEO)

https://www.youtube.com/watch?v=NYvxg8uDsAA

Keeping away the quality or not of the "Final Fusion", what is really concerning are the jumps that appear when MERGE is done, this is not a video render glitch, i can already see the jumps just scrolling trough the images in the MERGE folder.
So there is something that's not working like should i think.

Here you have the A folder with the base frames video, ALIGNED images and MERGED ones, so you can see that jumps start happening when do the MERGE:

https://drive.google.com/open?id=10rBAnx9KZ8wxa1PsrImDZuhfquSWgA7O

minImages Training Variable

I saw that by default dfaker will max out the used images for a given data set at 2000. My A set is under 2000 while my B set is around 12000. Is this code minImages = 2000 by design to only take in 2000 at a time and it randomizes them? When the script first loads it doesn't just load the first 2000 but all of the images, only the output afterward shows that it's maxing at 2000. Does the script choose the best 2000 to use?

If it doesn't, would it be possible to set it to use every photo from each data set? So for example, using 1500 from A and 6204 from B?
When I use the commented extra like this minImages = min(len(images_A),len(images_B))*20, it fails to run.

Error

Save model weights
Traceback (most recent call last):
File "train.py", line 148, in
figure = figure.reshape( (4,4) + figure.shape[1:] )
ValueError: cannot reshape array of size 884736 into shape (4,4,3,128,128,3)

Face area in conversion

Hi,
First of all, great job and thanks for sharing

I want to change the face area in merging process. The chin of the two faces are different height, in training preview the shape correction is right but is not carried to the conversion
Can I change any part of the code to modify the conversion area?

Thank you