hangz-nju-cuhk / rotate-and-render

Code for Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR 2020)

Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR 2020)

Hang Zhou*, Jihao Liu*, Ziwei Liu, Yu Liu, and Xiaogang Wang.

[Paper] [Video]

We propose a novel un-supervised framework that can synthesize photorealistic rotated faces using only single-view image collections in the wild. Our key insight is that rotating faces in the 3D space back and forth, and re-rendering them to the 2D plane can serve as a strong self-supervision.

Requirements

• Python 3.6 is used. Basic requirements are listed in the 'requirements.txt'.

pip install -r requirements.txt

• Install the Neural_Renderer following the instructions.

• Download checkpoint and BFM model from ckpt_and_bfm.zip， put it in 3ddfa and unzip it. Our 3D models are borrowed from 3DDFA.

DEMO

Rotate faces with examples provided

1. Download the checkpoint and put it in ./checkpoints/rs_model.

2. Run a simple Rotate-and-Render demo, the inputs are stored at 3ddfa/example.

• Modify experiments/v100_test.sh, the --poses are the desired degrees (range -90 to 90), choose 0 as frontal face.

• Run bash experiments/v100_test.sh, results will be saved at ./results/.

DEVELOP

Prepare your own dataset for testing and training.

Preprocessing

1. Save the 3D params of human faces to 3ddfa/results by 3ddfa.

cd 3ddfa
python inference.py --img_list example/file_list.txt --img_prefix example/Images --save_dir results
cd ..

Data Preparation

Modify class dataset_info() inside data/__ini__.py, then prepare dataset according to the pattern of the existing example. You can add the information about a new dataset to each instance of the class.

1. prefix The absolute path to the dataset.
2. file_list The list of all images, the absolute path could be incorrect as it is defined in the prefix
3. land_mark_list The list that stores all landmarks of all images.
4. params_dir the path that stores all the 3D params processed before.
5. dataset_names the dictionary that maps dataset NAMEs to their information. This is used in the parsers as --dataset NAME.
6. folder_level the level of folders from the prefix to images (.jpgs). For example the folder_level is 2 if a image is stored as prefix/label/image.jpg.

Training and Inference

• Modify experiments/train.sh and use bash experiments/train.sh for training on new datasets.

• Visualization is supported during training with Tensorboard.

• Please see the DEMO part for inference.

Details of the Models

We provide two models with trainers in this repo, namely rotate and rotatespade. The "rotatespade" model is an upgraded one which is different from that described in our paper. A conditional batchnorm module is added according to landmarks predicted from the 3D face. Our checkpoint is trained on this model. We have briefly described this model in our supplementary materials.

License and Citation

The usage of this software is under CC-BY-4.0.

@inproceedings{zhou2020rotate,
  title     = {Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images},
  author    = {Zhou, Hang and Liu, Jihao and Liu, Ziwei and Liu, Yu and Wang, Xiaogang},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2020},
}

Acknowledgement

• The structure of this codebase is borrowed from SPADE.
• The SyncBN module is used in our code.
• We directly borrow the 3DDFA implementation for 3D reconstruction.

rotate-and-render's Issues

crash when run v100_test.sh

pytorch 1.2.0 two gpu cards. err msg "
terminate called after throwing an instance of 'c10::Error' what(): CUDA error: driver shutting down

render_thread and gpu model might have something wrong ?

3dffa inference

Hello, when I need 3dffa's inference.py to obtain 3D information, this sentence "Downloading the face detection CNN. Please wait..." appears, and there is no response afterwards, is there any solution? Hope to reply, thanks！

How can this be run with a single GPU?

I have attempted to limit the GPU IDs to 0 in both the script and code, but it results in a CUDA error. Do you have any advice on how inference may be run on a single GPU?

About MS1M-ArcFace+RR

Hi,Cloud you please opensource the MS1M-ArcFace datasets，I try the RR on MS1M-ArcFace and LFW,CALFW,CPLFW,CFP_FP,CFP_FF,VGG2_FP,AGEDB_30 and IJB-C, But the results is bad,even the convergence of training is harder then baseline,I check the images after RR, looks reasonable,hope to get answer

Can i use the model to change the pose(yaw pitch roll) of the face?

Noisy/broken output

I have previously used this network just fine, but recently when trying to use it, it sometimes produces output that doesn't make sense. I have attached one such image here:

The output is inconsistent based on the input (same image outputs different results), and on some runs the network seems to work well. This seems to affect the first images of a batch the most. Any one else encountered this kind of issue? I'm using an RTX2080TI (over ssh) and cuda 11.2

How to test v100_test.py without GPU ?

Is there any way to run the v100_test.py without GPU ?

Report Bugs

hi, I've downloaded the repo, run the code, and python reports the following errors:

1. /options/base_options.py", line 177, in parse
   (3 if opt.use_BG else 0)
   AttributeError: 'Namespace' object has no attribute 'use_BG'
   It seems opt.use_BG is not defined.

2. Set opt.use_BG to True manually, run again:
   /models/rotatespade_model.py", line 181, in initialize_networks
   netE = networks.define_E(opt) if opt.use_vae else None
   AttributeError: 'Namespace' object has no attribute 'use_vae'

So are these bugs or I run experiments/v100_test.sh in a wrong way?

Difficult to extend multi-pose synthesis to large scale dataset

Either cuda out of memory though 3 gpus, or no response printed on the terminal for a long time.

Add "pitch_poses" get error

When I add "--pitch_poses 0" at the end of "experiments/v100_test.sh", then run it, I got error like following:

----------------- Options ---------------
align: True [default: False]
aspect_ratio: 1.0
cache_filelist_read: False
cache_filelist_write: False
checkpoints_dir: ./checkpoints
chunk_size: [1] [default: None]
contain_dontcare_label: False
crop_size: 256
dataset: test_align [default: ms1m,casia]
dataset_mode: allface
device_count: 2 [default: 8]
display_winsize: 256
erode_kernel: 21
gpu_ids: 0,1 [default: 0]
heatmap_size: 2.5 [default: 3]
how_many: inf
init_type: xavier
init_variance: 0.02
isTrain: False [default: None]
label_mask: True [default: False]
label_nc: 5
landmark_align: False
list_end: 10 [default: inf]
list_num: 0
list_start: 0
load_from_opt_file: False
load_size: 256
max_dataset_size: 9223372036854775807
model: rotatespade [default: rotate]
multi_gpu: True [default: False]
nThreads: 3 [default: 1]
name: mesh2face
names: rs_model [default: rs_ijba3]
nef: 16
netG: rotatespade [default: rotate]
ngf: 64
no_flip: True
no_gaussian_landmark: True [default: False]
no_instance: True
no_pairing_check: False
norm_D: spectralinstance
norm_E: spectralinstance
norm_G: spectralsyncbatch [default: spectralinstance]
output_nc: 3
phase: test
pitch_poses: [0.0] [default: None]
posesrandom: False
preprocess_mode: scale_width_and_crop
render_thread: 1 [default: 2]
resnet_initial_kernel_size: 7
resnet_kernel_size: 3
resnet_n_blocks: 9
resnet_n_downsample: 4
results_dir: ./results/
save_path: ./results/
serial_batches: True
trainer: rotate
which_epoch: latest
yaw_poses: None
----------------- End -------------------
dataset [AllFaceDataset] of size 300 was created
Testing gpu [0]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
start prefetching data...
Process Process-1:
Traceback (most recent call last):
File "/home/zhuzhou/Software/anaconda3/envs/mvb/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/zhuzhou/Software/anaconda3/envs/mvb/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/zhuzhou/MyGit/Rotate-and-Render/data/data_utils.py", line 146, in prefetch_data
prefetcher = data_prefetcher(dataloader, opt, render_layer)
File "/home/zhuzhou/MyGit/Rotate-and-Render/data/data_utils.py", line 99, in init
self.preload()
File "/home/zhuzhou/MyGit/Rotate-and-Render/data/data_utils.py", line 124, in preload
self.next_input = get_multipose_test_input(data, self.render_layer, self.opt.yaw_poses, self.opt.pitch_poses)
File "/home/zhuzhou/MyGit/Rotate-and-Render/data/data_utils.py", line 68, in get_multipose_test_input
= render.rotate_render(data['param_path'], real_image, data['M'], pitch_pose=pose)
File "/home/zhuzhou/MyGit/Rotate-and-Render/models/networks/rotate_render.py", line 55, in rotate_render
yaw_pose=yaw_pose, pitch_pose=pitch_pose)
File "/home/zhuzhou/MyGit/Rotate-and-Render/models/networks/render.py", line 364, in _forward
large_pose=large_pose, yaw_pose=yaw_pose, pitch_pose=pitch_pose)
File "/home/zhuzhou/MyGit/Rotate-and-Render/models/networks/render.py", line 303, in generate_vertices_and_rescale_to_img
yaw_pose=yaw_pose, pitch_pose=pitch_pose)
File "/home/zhuzhou/MyGit/Rotate-and-Render/models/networks/render.py", line 185, in _parse_param
p = angle2matrix(angle) * s
File "/home/zhuzhou/MyGit/Rotate-and-Render/models/networks/render.py", line 98, in angle2matrix
Ry=np.array([[ cos(y), 0, sin(y)],
TypeError: must be real number, not NoneType

How can I run it correctly with yaw and pitch are all 0?
Thanks

Requirements about the training face dataset

Hello,
Thanks for your great works.
I want to know whether this model training has some requirements on the dataset such as dataset size, image scenes, data prepocessing method.
Looking forward to your reply.

How to preprocess the CelebA-HQ dataset?

Thanks for your sharing. I saw your paper mention that "Our results are cropped for better visualization. It is recommended to zoom-in for a better view. " May I ask how you crop the CelebA-HQ? Is the process the same with MultiPIE? is it 128 * 128? Thank you very much!

question about to prepare the dataset

i run the inference.py to generate the realign_lmk of images as you provided in the example file, but it run like this error, how can i solve it, please.

/Rotate-and-Render-master/3ddfa/inference.py
0%| | 0/8 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/yeluyue/dl/Rotate-and-Render-master/3ddfa/inference.py", line 168, in
main(args)
File "/home/yeluyue/dl/Rotate-and-Render-master/3ddfa/inference.py", line 84, in main
pts_2d_5 = get_5lmk_from_68lmk(pts_2d_68)
File "/home/yeluyue/dl/Rotate-and-Render-master/3ddfa/utils/inference.py", line 14, in get_5lmk_from_68lmk
left_eye = lmk68[36:42, :].mean(axis=0)
IndexError: too many indices for array

How to add roll pose？？

Render time

Hi, thanks for your good work!
I tested it, the render time is about 1s, right? it is too slow, can be improved?

error when run v100_test.sh

dataset [AllFaceDataset] of size 8 was created
Testing gpu [0]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
start prefetching data...
Process Process-1:
Traceback (most recent call last):
File "/ssd/anaconda3/envs/rr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/ssd/anaconda3/envs/rr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/ssd/experiment/Rotate-and-Render/data/data_utils.py", line 146, in prefetch_data
prefetcher = data_prefetcher(dataloader, opt, render_layer)
File "/ssd/experiment/Rotate-and-Render/data/data_utils.py", line 99, in init
self.preload()
File "/ssd/experiment/Rotate-and-Render/data/data_utils.py", line 124, in preload
self.next_input = get_multipose_test_input(data, self.render_layer, self.opt.yaw_poses, self.opt.pitch_poses)
File "/ssd/experiment/Rotate-and-Render/data/data_utils.py", line 65, in get_multipose_test_input
= render.rotate_render(data['param_path'], real_image, data['M'], yaw_pose=pose)
File "/ssd/experiment/Rotate-and-Render/models/networks/rotate_render.py", line 80, in rotate_render
rendered_images, depths, masks, = self.renderer(vertices_ori_normal, self.faces_use, texs) # rendered_images: batch * 3 * h * w, masks: batch * h * w
File "/ssd/anaconda3/envs/rr/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/ssd/anaconda3/envs/rr/lib/python3.6/site-packages/neural_renderer-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/renderer.py", line 68, in forward
return self.render(vertices, faces, textures)
File "/ssd/anaconda3/envs/rr/lib/python3.6/site-packages/neural_renderer-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/renderer.py", line 148, in render
vertices = nr.look(vertices, self.eye, self.camera_direction)
File "/ssd/anaconda3/envs/rr/lib/python3.6/site-packages/neural_renderer-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/look.py", line 41, in look
if up.ndimension() == 1:
AttributeError: 'NoneType' object has no attribute 'ndimension'

pytorch 1.3.1 torchvison4.2 undefined symbol: _ZN2at19UndefinedTensorImpl10_singletonE

ImportError: /data1/visiondection/3dface/Rotate-and-Render-master/Rotate-and-Render-master/envRote/lib/python3.6/site-packages/neural_renderer/cuda/load_textures.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at19UndefinedTensorImpl10_singletonE

neural_renderer needs torch version maybe isnot right

inference.py error

About the roll pose

I try to add roll_poses, and do the same things as add pitch_poses, But It didn't work. How to fix it?

add an example in colab.

Please can add a demo in Colab, to test with my images

how to replace 3ddfa_v2 with 3ddfa ?

3ddfa_v2 is much accuracy than 3ddfa on face fitting

Run demo error

Hi,
Thanks for your great project,
My environment is torch 1.1.0 and neural-renderer-pytorch 1.1.3,
I call library to check torch GPU is work and return" torch.cuda.is_available()" is True.
When I run demo, I have only one gpu, and change the experiments/v100_test.sh as
python -u test_multipose.py
--names rs_model
--dataset example
--list_start 0
--list_end 10
--dataset_mode allface
--gpu_ids 0
--netG rotatespade
--norm_G spectralsyncbatch
--model rotatespade
--label_nc 5
--nThreads 1
--heatmap_size 2.5
--chunk_size 1
--no_gaussian_landmark
--multi_gpu
--device_count 1
--render_thread 1
--label_mask
--align
--erode_kernel 21
--yaw_poses 0 30
and revise test_multipose.py#L102 as opt.gpu_ids = [0]
But still got the error as following:

----------------- Options ---------------
align: True [default: False]
aspect_ratio: 1.0
cache_filelist_read: False
cache_filelist_write: False
checkpoints_dir: ./checkpoints
chunk_size: [1] [default: None]
contain_dontcare_label: False
crop_size: 256
dataset: example [default: ms1m,casia]
dataset_mode: allface
device_count: 1 [default: 8]
display_winsize: 256
erode_kernel: 21
gpu_ids: 0
heatmap_size: 2.5 [default: 3]
how_many: inf
init_type: xavier
init_variance: 0.02
isTrain: False [default: None]
label_mask: True [default: False]
label_nc: 5
landmark_align: False
list_end: 10 [default: inf]
list_num: 0
list_start: 0
load_from_opt_file: False
load_size: 256
max_dataset_size: 9223372036854775807
model: rotatespade [default: rotate]
multi_gpu: True [default: False]
nThreads: 1
name: mesh2face
names: rs_model [default: rs_ijba3]
nef: 16
netG: rotatespade [default: rotate]
ngf: 64
no_flip: True
no_gaussian_landmark: True [default: False]
no_instance: True
no_pairing_check: False
norm_D: spectralinstance
norm_E: spectralinstance
norm_G: spectralsyncbatch [default: spectralinstance]
output_nc: 3
phase: test
pitch_poses: None
posesrandom: False
preprocess_mode: scale_width_and_crop
render_thread: 1 [default: 2]
resnet_initial_kernel_size: 7
resnet_kernel_size: 3
resnet_n_blocks: 9
resnet_n_downsample: 4
results_dir: ./results/
save_path: ./results/
serial_batches: True
trainer: rotate
which_epoch: latest
yaw_poses: [0.0, 30.0] [default: None]
----------------- End -------------------
dataset [AllFaceDataset] of size 8 was created
render_gpu_ids [0]
Testing gpu [0]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
start prefetching data...
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
(************* each image render time: 8.694 *****************)
/home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:117
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: driver shutting down (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:556)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fb0e533f441 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fb0e533ed7a in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: + 0x1390c (0x7fb091a7990c in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: torch::CudaIPCSentData::~CudaIPCSentData() + 0x215 (0x7fb0e5674115 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x11e288 (0x7fb0e5676288 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x430f1 (0x7fb0ea2190f1 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x431ea (0x7fb0ea2191ea in /lib/x86_64-linux-gnu/libc.so.6)
frame #7: + 0x20fad9 (0x55e62e3ebad9 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)
frame #8: + 0x20fbb8 (0x55e62e3ebbb8 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)
frame #9: PyErr_PrintEx + 0x32 (0x55e62e3ebc22 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)
frame #10: PyRun_SimpleStringFlags + 0x66 (0x55e62e3f1f96 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)
frame #11: Py_Main + 0x423 (0x55e62e3f5d73 in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)
frame #12: main + 0xee (0x55e62e2bff2e in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)
frame #13: __libc_start_main + 0xe7 (0x7fb0ea1f7b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #14: + 0x1c327f (0x55e62e39f27f in /home/infor/anaconda3/envs/python36_RotateRender_torch1.4/bin/python)

in the end there are no result images saved into that folder.
can you give me some suggestions? Thank you so much.

error when run demo

Testing gpu [0, 1, 2, 3, 4, 5, 6]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
start prefetching data...
Process Process-1:
Traceback (most recent call last):
File "/home/yangchuqiao/.conda/envs/render/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/yangchuqiao/.conda/envs/render/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/yangchuqiao/code/github/Rotate-and-Render/data/data_utils.py", line 146, in prefetch_data
prefetcher = data_prefetcher(dataloader, opt, render_layer)
File "/home/yangchuqiao/code/github/Rotate-and-Render/data/data_utils.py", line 99, in init
self.preload()
File "/home/yangchuqiao/code/github/Rotate-and-Render/data/data_utils.py", line 124, in preload
self.next_input = get_multipose_test_input(data, self.render_layer, self.opt.yaw_poses, self.opt.pitch_poses)
File "/home/yangchuqiao/code/github/Rotate-and-Render/data/data_utils.py", line 65, in get_multipose_test_input
= render.rotate_render(data['param_path'], real_image, data['M'], yaw_pose=pose)
File "/home/yangchuqiao/code/github/Rotate-and-Render/models/networks/rotate_render.py", line 100, in rotate_render
tex_a_crop = self.get_render_from_vertices(rendered_images_erode[n], vertices_in_ori_img[n])
File "/home/yangchuqiao/code/github/Rotate-and-Render/models/networks/render.py", line 340, in get_render_from_vertices
textures = img_ori[vertices_in_ori_img[1, :].round().clamp(0, h - 1).long(),
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

What is the purpose of the image saved along with the text in the 3ddfa?

i noticed when running the inference.py we have a result saved into results/NAME_OF_FILE.txt and results/NAME_OF_FILE.jpg

but the thing is, when running the v100_test.sh and changing the image name in example/imlist to the image i want to rotate, i noticed it only searches in the results folder for the txtfile of landmarks but doesnot use the image (saved there) at all.

but uses the image saved at the example folder.

so my question is, what is the purpose of that specific output (the image) of the inference?

no kernel image is available for the execution of device

I installed pytorch 1.2 torchvision 0.4 for the neural_renderer but when I ran the v100_test.sh file, it said no kernel image is available for the execution of device. I searched online & it appeared that it is because of the cuda torch compatibility issue but when I upgraded my pytorch version, the neural_renderer doesn't work. I'm stuck, can you help me with this? I'm attaching my error output here as well.

AttributeError: 'Namespace' object has no attribute 'face_vgg'

python inference.py IndexError: too many indices for array

When I run the inference.py, the error occurs. Please help. Thanks very much.

python inference.py --img_list example/file_list.txt --img_prefix example/Images --save_dir results

/Rotate-and-Render/3ddfa$ python inference.py --img_list example/file_list.txt --img_prefix example/Images --save_dir results
0%| | 0/8 [00:04<?, ?it/s]
Traceback (most recent call last):
File "inference.py", line 168, in
main(args)
File "inference.py", line 84, in main
pts_2d_5 = get_5lmk_from_68lmk(pts_2d_68)
File "/home/zq/workspace/Rotate-and-Render/3ddfa/utils/inference.py", line 14, in get_5lmk_from_68lmk
left_eye = lmk68[36:42, :].mean(axis=0)
IndexError: too many indices for array

How to use your trained model to rotate other face data sets？

If it can be run with a single GPU on Windows?

Yes, it can! I have solved it, thank you!

v100_test.py with wrong result

As i have only one GPU, i changed the v100_test.py, but result seems failed.
Below is the log, and the jpg files in result folder are wrong, all are black.

##############################################################
./experiments/v100_test.sh
----------------- Options ---------------
align: True [default: False]
aspect_ratio: 1.0
cache_filelist_read: False
cache_filelist_write: False
checkpoints_dir: ./checkpoints
chunk_size: [1] [default: None]
contain_dontcare_label: False
crop_size: 256
dataset: example [default: ms1m,casia]
dataset_mode: allface
device_count: 2 [default: 8]
display_winsize: 256
erode_kernel: 21
gpu_ids: 0,1 [default: 0]
heatmap_size: 2.5 [default: 3]
how_many: inf
init_type: xavier
init_variance: 0.02
isTrain: False [default: None]
label_mask: True [default: False]
label_nc: 5
landmark_align: False
list_end: 10 [default: inf]
list_num: 0
list_start: 0
load_from_opt_file: False
load_size: 256
max_dataset_size: 9223372036854775807
model: rotatespade [default: rotate]
multi_gpu: True [default: False]
nThreads: 1
name: mesh2face
names: rs_model [default: rs_ijba3]
nef: 16
netG: rotatespade [default: rotate]
ngf: 64
no_flip: True
no_gaussian_landmark: True [default: False]
no_instance: True
no_pairing_check: False
norm_D: spectralinstance
norm_E: spectralinstance
norm_G: spectralsyncbatch [default: spectralinstance]
output_nc: 3
phase: test
pitch_poses: None
posesrandom: False
preprocess_mode: scale_width_and_crop
render_thread: 1 [default: 2]
resnet_initial_kernel_size: 7
resnet_kernel_size: 3
resnet_n_blocks: 9
resnet_n_downsample: 4
results_dir: ./results/
save_path: ./results/
serial_batches: True
trainer: rotate
which_epoch: latest
yaw_poses: [0.0] [default: None]
----------------- End -------------------
dataset [AllFaceDataset] of size 8 was created
Testing gpu [0]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
start prefetching data...
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
(************* each image render time: 13.314 ****)
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
/home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
process image..../results/rs_model/example/orig/yaw_0.0_Ann_Veneman_0010.jpg
processed num 1
( each image time total: 14.036 ****)
( each image render time: 0.001 ****)
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
Error in forward_face_index_map_1: invalid device function
Error in forward_face_index_map_2: invalid device function
Error in forward_texture_sampling: invalid device function
process image..../results/rs_model/example/orig/yaw_0.0_Benjamin_Netanyahu_0005.jpg
processed num 2
( each image time total: 0.170 ****)
( each image render time: 0.036 ****)
process image..../results/rs_model/example/orig/yaw_0.0_Hugo_Chavez_0033.jpg
processed num 3
( each image time total: 0.188 ****)
( each image render time: 0.074 *****************)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: driver shutting down (insert_events at /opt/conda/conda-bld/pytorch_1579022034529/work/c10/cuda/CUDACachingAllocator.cpp:756)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7fbd76b71627 in /home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: + 0x1af78 (0x7fbd76db1f78 in /home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: + 0x1cbd1 (0x7fbd76db3bd1 in /home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: torch::CudaIPCSentData::~CudaIPCSentData() + 0x241 (0x7fbd776694b1 in /home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x5346c5 (0x7fbd7766b6c5 in /home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x534b8d (0x7fbd7766bb8d in /home/forest/anaconda3/envs/python361/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x39ff8 (0x7fbd7b2eeff8 in /lib/x86_64-linux-gnu/libc.so.6)
frame #7: + 0x3a045 (0x7fbd7b2ef045 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: + 0x20fad9 (0x5592f0444ad9 in /home/forest/anaconda3/envs/python361/bin/python)
frame #9: + 0x20fbb8 (0x5592f0444bb8 in /home/forest/anaconda3/envs/python361/bin/python)
frame #10: PyErr_PrintEx + 0x32 (0x5592f0444c22 in /home/forest/anaconda3/envs/python361/bin/python)
frame #11: PyRun_SimpleStringFlags + 0x66 (0x5592f044af96 in /home/forest/anaconda3/envs/python361/bin/python)
frame #12: Py_Main + 0x423 (0x5592f044ed73 in /home/forest/anaconda3/envs/python361/bin/python)
frame #13: main + 0xee (0x5592f0318f2e in /home/forest/anaconda3/envs/python361/bin/python)
frame #14: __libc_start_main + 0xf0 (0x7fbd7b2d5830 in /lib/x86_64-linux-gnu/libc.so.6)
frame #15: + 0x1c327f (0x5592f03f827f in /home/forest/anaconda3/envs/python361/bin/python)

process image..../results/rs_model/example/orig/yaw_0.0_Julianne_Moore_0012.jpg
processed num 4
(************* each image time total: 0.254 ****)
( each image render time: 0.042 ****)
process image..../results/rs_model/example/orig/yaw_0.0_Keanu_Reeves_0010.jpg
processed num 5
( each image time total: 0.187 ****)
( each image render time: 0.002 ****)
process image..../results/rs_model/example/orig/yaw_0.0_Norah_Jones_0003.jpg
processed num 6
( each image time total: 0.146 ****)
( each image render time: 0.001 ****)
process image..../results/rs_model/example/orig/yaw_0.0_Robin_Wright_Penn_0001.jpg
processed num 7
( each image time total: 0.152 ****)
( each image render time: 0.037 ****)
process image..../results/rs_model/example/orig/yaw_0.0_Vitali_Klitschko_0003.jpg
processed num 8
( each image time total: 0.200 *****************)
finished
#########################################################################

一些建议some advice。

H：
hello！

test_multipose.py:

ngpus = opt.device_count
render_gpu_ids = list(range(ngpus - opt.render_thread, ngpus))
render_layer_list = []
for gpu in render_gpu_ids:
    opt.gpu_ids = gpu
    render_layer = TestRender(opt)
    render_layer_list.append(render_layer)

opt.gpu_ids = list(range(0, ngpus - opt.render_thread))？？？？？？？？？？？？？？？

这句有问题，如果device_count=1,ngpus就等于1，这时候，opt.render_thread（你默认为2）假设等于1呢？这个时候opt.gpu_ids = list(range(0, ngpus - opt.render_thread)) 就等于【】，没有值的空表。
so：opt.gpu_ids = list(range(0, ngpus)才能继续下去。

我觉得作者的码基于自身8个gpu，多线程的设备环境。
你可以在read me说明清晰。
另外，眼前，一个项目的流行，进一步的发展很大程度上依赖于colab，paddle等云端平台，开发者不会天天conda的，大家习惯，先扔进云端走一遍。
作者可以自己先走一遍，这样对于后续有极大的好处。
也可以让你避免代码中出现你之前的use_BG、
use_VAE之类改动而忘记删码的不必要的麻烦，
这些问题会导致明明挺优秀的项目无法流行，这么大量的参数，没有说明，连test都无法顺利向前走（不是大家走不过去，而是要花时间），会很可惜的。

祝顺利！

There is a problem with this sentence. If device_count = 1, ngpus is equal to 1, at this time, suppose that opt.render_thread (you default is 2) is equal to 1? At this time opt.gpu_ids = list (range (0, ngpus-opt.render_thread)) is equal to [], an empty table with no value.

so: opt.gpu_ids = list (range (0, ngpus) to continue.

I think the author's code is based on its 8 gpu, multi-threaded device environment.

You can explain clearly in read me.

In addition, in front of us, the popularity and further development of a project relies heavily on cloud platforms such as colab and paddle. Developers will not conda every day. Everyone is used to throwing them into the cloud first.

The author can go through it by himself first, which has great benefits for the follow-up.

It also allows you to avoid your previous use_BG,

Use_VAE and other changes and forget the unnecessary trouble of deleting the code,

These problems will lead to the failure of popular projects that are clearly outstanding. With so many parameters and no description, even the test cannot go forward smoothly (not everyone can't go through it, but it takes time), it will be a pity.

Good luck

Problem with training: successfully opened dynamic library libcudart.so.10.1

Greetings,

I am trying to train on my own custom dataset. When i start the train script I get the following printed and nothing happens further.

I tensorflow/stream_executor/platform/default/dso_loader.cc:48] successfully opened dynamic library libcudart.so.10.1

Am i missing something anywhere? P.s. I could run the test script for inference and it worked just fine.

Thank you

the effect is not very good

use my own data,the effect is not very good

能否提供D和D_rotate的pretrain model？

感谢您杰出的工作，请问能否提供D和D_rotate的pretrain model呢，谢谢。

running env?

Hi, what is the env i.e. python version, it runs with? The link to the neural renderer is python2 and relies on old pytorch and cuda8, which seems conflict with this repo's requirements. Are you using a 3rd party neural renderer ?

ModuleNotFoundError: No module named 'data.single_dataset'

if run "experiments/test.sh", it will be set "dataset_mode single=single", but there is no file "single_dataset.py" in this folder of data!

More information of the arguments.

Hi there, i am so interested in your work, and try to re-train my own data. But there are too many arguments in your code without specific comments or explaining in your paper, and it is a little confused to understand and re-set them. could you provide more information of the arguments?

Pytorch queue is stuck

While running the train script, the dataloader gets stuck at data = training_queue.get(block=True) in train.py

OS: Linux 18.04
#GPU: 4
Pytorch: 1.7.1 cuda
CUDA: 11.0
cuDNN: 8.2

ValueError: not enough values to unpack (expected 3, got 1

Thanks for sharing the project,I meet some problems in your code!
(py37) ty@ty-HP-Z8-G4-Workstation:~/文档/人脸识别/Rotate-and-Render-master$ bash experiments/v100_test.sh
----------------- Options ---------------
align: True [default: False]
aspect_ratio: 1.0
cache_filelist_read: False
cache_filelist_write: False
checkpoints_dir: ./checkpoints
chunk_size: [1] [default: None]
contain_dontcare_label: False
crop_size: 256
dataset: example [default: ms1m,casia]
dataset_mode: allface
device_count: 1 [default: 8]
display_winsize: 256
erode_kernel: 21
gpu_ids: 0,1 [default: 0]
heatmap_size: 2.5 [default: 3]
how_many: inf
init_type: xavier
init_variance: 0.02
isTrain: False [default: None]
label_mask: True [default: False]
label_nc: 5
landmark_align: False
list_end: 10 [default: inf]
list_num: 0
list_start: 0
load_from_opt_file: False
load_size: 256
max_dataset_size: 9223372036854775807
model: rotatespade [default: rotate]
multi_gpu: True [default: False]
nThreads: 3 [default: 1]
name: mesh2face
names: rs_model [default: rs_ijba3]
nef: 16
netG: rotatespade [default: rotate]
ngf: 64
no_flip: True
no_gaussian_landmark: True [default: False]
no_instance: True
no_pairing_check: False
norm_D: spectralinstance
norm_E: spectralinstance
norm_G: spectralsyncbatch [default: spectralinstance]
output_nc: 3
phase: test
pitch_poses: None
posesrandom: False
preprocess_mode: scale_width_and_crop
render_thread: 1 [default: 2]
resnet_initial_kernel_size: 7
resnet_kernel_size: 3
resnet_n_blocks: 9
resnet_n_downsample: 4
results_dir: ./results/
save_path: ./results/
serial_batches: True
trainer: rotate
which_epoch: latest
yaw_poses: [0.0, 30.0] [default: None]
----------------- End -------------------
dataset [AllFaceDataset] of size 8 was created
Testing gpu [0]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
start prefetching data...
Process Process-1:
Traceback (most recent call last):
File "/home/ty/anaconda3/envs/py37/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/ty/anaconda3/envs/py37/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/ty/文档/人脸识别/Rotate-and-Render-master/data/data_utils.py", line 146, in prefetch_data
prefetcher = data_prefetcher(dataloader, opt, render_layer)
File "/home/ty/文档/人脸识别/Rotate-and-Render-master/data/data_utils.py", line 99, in init
self.preload()
File "/home/ty/文档/人脸识别/Rotate-and-Render-master/data/data_utils.py", line 124, in preload
self.next_input = get_multipose_test_input(data, self.render_layer, self.opt.yaw_poses, self.opt.pitch_poses)
File "/home/ty/文档/人脸识别/Rotate-and-Render-master/data/data_utils.py", line 65, in get_multipose_test_input
= render.rotate_render(data['param_path'], real_image, data['M'], yaw_pose=pose)
File "/home/ty/文档/人脸识别/Rotate-and-Render-master/models/networks/rotate_render.py", line 80, in rotate_render
rendered_images, depths, masks, = self.renderer(vertices_ori_normal, self.faces_use, texs) # rendered_images: batch * 3 * h * w, masks: batch * h * w
ValueError: not enough values to unpack (expected 3, got 1)

inference slowly

When I experence this model，the speed of inference is too slowly at CelebA dataset. How can I do it?

Cant run test

Get error:

FileNotFoundError: [Errno 2] No such file or directory: '3ddfa/train.configs\texMU.npy'

donwloaded train.configs from 3ddfa repo. it doesnt contain texMU.npy file

File missing error : 106_index.npy

On running bash experiments/v100_test.sh, got the following error:

Traceback (most recent call last):
  File "test_multipose.py", line 100, in <module>
    render_layer = TestRender(opt)
  File "/content/Rotate-and-Render/models/networks/rotate_render.py", line 23, in __init__
    self.keypoints_106 = _load(osp.join(self.d, '106_index.npy'))
  File "/content/Rotate-and-Render/models/networks/rotate_render.py", line 14, in _load
    return np.load(fp)
  File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 428, in load
    fid = open(os_fspath(file), "rb")
FileNotFoundError: [Errno 2] No such file or directory: './3ddfa/train.configs/106_index.npy'```

How to fix?
Thanks in advance

how to train with my data

hello ,author.Thanks your code,but I have some questions about train .I use CASIA dataset to train GAN ,but I get some problems.Can you tell me how to deal with trainning dataset?

How to reduce gpus to train?

Hello! Thank you for your splendid work! But I have some problem on training. I know you train your model with 8 gpus, however, I only have 3 gpus. Could you tell me how to reduce gpus for training! I will appreciate it if you could give me your advise!

Run bash/examples/train.sh, the loss value is incorrect

Follow the steps in the readme.md file and use your own data set for training. D_Fake and D_Real are always 0, and the generated results are gray images
similar to：
(epoch: 2, iters: 74, time: 3.480) GAN: 3.844 GAN_Feat: 30.297 VGG: 6.472 D_Fake: 0.000 D_real: 0.000
What changes do I need to make

The hair color of the output image is different from that of the original image

I tried to run V100.sh。 However, the hair color of the face in the output image is different from the original image. I want to know how to solve this problem.Thank you

Error in weight type and Input type

Hello,

I wanted to train the rotatespade model for my custom data, but I get the following error

Can someone please help me at this stage?

CelebA-HQ results

Hi, the CelebA-HQ results on paper are quite impressive!
Yet I tried some CelebA-HQ(1024*1024) images, which are selected randomly, there are some checkboard artefacts, please refer to the attachments.

I think the gan model is for 256 resolution, do you use another model for those high resolution images?
Also, any suggestions on setting parameters or examples would be appreciate.

Thanks!

Pretrained network G has fewer layers

When I run v100_test.sh using the pretrained G，some of the information is
Pretrained network G has fewer layers; The following are not initialized:
['resnet_layers0', 'resnet_layers1', 'resnet_layers2', 'resnet_layers3', 'resnet_layers4', 'resnet_layers5', 'resnet_layers6', 'resnet_layers7', 'resnet_layers8']
Is the pretrained G model incomplete or the paramenters of the resnet can be initialled randomly?

large pose result is weird

Thanks for release your job！And I meet a problem now.
I have some profile face images, and want to get the frontal image. The results of little pose ( less than 60 ) are nice , but for large pose, such as 70° or 90°，the results are bad.
Is it because the image resolution is too low? (128*128）Waiting for your reply, thanks!!