barisgecer / ostec Goto Github PK

Thank you very much for sharing such excellent work. I want to know how to obtain the visibility mask of the raw face image. Thank you again! Looking forward to your reply!

Hello, some input images have such an error, do you know why? Hope to get an answer, thank you very much!

Traceback (most recent call last):
File "run_ostec.py", line 98, in
main(args)
File "run_ostec.py", line 73, in main
final_uv, results_dict = operator.run(img, fitting, face_mask)
File "/data/code/OSTeC/core/operator.py", line 351, in run
face = self.get_tmesh(im, reconstruction_dict, face_mask)
File "/data/code/OSTeC/core/operator.py", line 207, in get_tmesh
im_masked = fill_UV(im_PIL2menpo(im_masked))
File "/data/code/OSTeC/utils/utils.py", line 132, in fill_UV
result = interp(np.ravel(xx[mask]), np.ravel(yy[mask]))
File "/data/anaconda3/envs/ostec/lib/python3.6/site-packages/scipy/interpolate/ndgriddata.py", line 79, in call
return self.values[i]
IndexError: index 0 is out of bounds for axis 0 with size 0

Hi!
Loved your work.
Any plans for releasing the code?

Thanks

when I run the command: python run_ostec.py --source_dir MyTestData/Input --save_dir MyTestData/Output -i 100 -m soft
occur a error : nvcc fatal : Host compiler targets unsupported OS.
I don't know how to solve it

https://drive.google.com/file/d/1TBoNt55vleRkMZaT9XKt6oNQmo8hkN-Q/view?usp=sharing

I found that the configuration file should be exported from Linux. Who can provide the configuration file suitable for Windows? Thank you very much.

Thanks for your great work。
I would like to know how to get the de-occluded frontal face image.
I input the occlusion image, and finally got the obj file and uv map. But there is no picture after de-occlusion

input:

render results after reconstruction:

code:

# fitting
fitting = deep3dmodel.recontruct(im_menpo2PIL(img), lms, test=False)

# Prepare Textured Trimesh with visible part of the face
face = operator.get_tmesh(im, fitting)

# Render to specific angle
from utils.image_rasterization import rasterize_image

im, _ = rasterize_image(face.tmesh, operator.img_shape, pose_angle_deg=[0, 0, 0], cam_dist=4.5)
cv2.imwrite(f'./split_test/ffhq_render_angle_0.jpg', np.array(im_menpo2PIL(im))[:, :, ::-1])

im, _ = rasterize_image(face.tmesh, operator.img_shape, pose_angle_deg=[0, 45, 0], cam_dist=4.5)
cv2.imwrite(f'./split_test/ffhq_render_angle_45.jpg', np.array(im_menpo2PIL(im))[:, :, ::-1])

I want to observe the reconstructed performance in meshlab or blender, please give me a tip how to generate the obj file, thank a lot.

Thanks for your great job. I

1. When I run the code it tips -f unrecongnized arguments.
2. Another problem is that I can not see any differences from the output datas between the different arguments with hard or soft.
3. Could you give some suggestions about how to handle with multiple images

python run_ostec.py --source_dir /content/9.jpg --save_dir /content/out [-f] -i 200 -m hard

/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
Traceback (most recent call last):
  File "run_ostec.py", line 21, in <module>
    from external.deep3dfacerecon.ostec_api import Deep3dModel
  File "/content/OSTeC/external/deep3dfacerecon/ostec_api.py", line 5, in <module>
    from models import create_model
  File "external/deep3dfacerecon/models/__init__.py", line 22, in <module>
    from models.base_model import BaseModel
  File "external/deep3dfacerecon/models/base_model.py", line 9, in <module>
    from . import networks
  File "external/deep3dfacerecon/models/networks.py", line 18, in <module>
    from .arcface_torch.backbones import get_model
ModuleNotFoundError: No module named 'models.arcface_torch'

I have NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 installed on the server. I followed the instructions to install the repository by installing cuda 10.0, cudnn 7.4 and tensorflow 1.14 inside ostec environment. But after all the models get loaded, I get this error while processing images.

Started: ./test_imgs/0000_0099_00.png Traceback (most recent call last): File "run_ostec.py", line 96, in <module> main(args) File "run_ostec.py", line 66, in main fitting = deep3dmodel.recontruct(im_menpo2PIL(img), lms) File "/home/suvidha/av-project/OSTeC/external/deep3dfacerecon/ostec_api.py", line 68, in recontruct coeffs = self.model.net_recon(self.model.input_img) File "/home/suvidha/anaconda3/envs/ostec/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "external/deep3dfacerecon/models/networks.py", line 98, in forward x = self.backbone(x) File "/home/suvidha/anaconda3/envs/ostec/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "external/deep3dfacerecon/models/networks.py", line 375, in forward return self._forward_impl(x) File "external/deep3dfacerecon/models/networks.py", line 358, in _forward_impl x = self.conv1(x) File "/home/suvidha/anaconda3/envs/ostec/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/suvidha/anaconda3/envs/ostec/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 423, in forward return self._conv_forward(input, self.weight) File "/home/suvidha/anaconda3/envs/ostec/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 420, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: CUDA error: no kernel image is available for execution on the device

Please help me with the solution. I am stuck and unable to make this repository work since 2 days now.
I also can't find cuda path (installed using conda) which I can export in bashrc.

is anyone meet this problem?
I install tensorflow 1.15.5 on Ubuntu, CUDA 11.1, cudnn 8
In file included from .conda/envs/ostecP/lib/python3.6/site-packages/tensorflow_core/include/third_party/eigen3/Eigen/Core:1,
from .conda/envs/ostecP/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/lib/strings/strcat.h:29,
from .conda/envs/ostecP/lib/python3.6/site-packages/tensorflow_core/include/tensorflow/core/lib/core/errors.h:24,
from ~/OSTeC/external/stylegan2/dnnlib/tflib/ops/tensorflow/core/framework/op.h:26,
from external/stylegan2/dnnlib/tflib/ops/fused_bias_act.cu:9:
.conda/envs/ostecP/lib/python3.6/site-packages/tensorflow_core/include/Eigen/Core:28:12: fatal error: cuda/std/complex: No such file or directory
28 | #include <cuda/std/complex>
| ^~~~~~~~~~~~~~~~~~
compilation terminated.

I keep getting error because of args parameters is not defiend
can you share args parameters to start with?

First of all, thank you very much for sharing such excellent work! When I complete the installation, and run "python run_ostec.py --source_dir [source_dir] --save_dir [save_dir] [-f] -i [iterations (default 200)] -m [soft|hard|auto]", the error occurrs：

`Using TensorFlow backend.
/////miniconda3/envs/ostec/lib/python3.6/site-packages/kornia/augmentation/augmentation.py:1875: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead.
category=DeprecationWarning,
Traceback (most recent call last):
File "/////OSTeC/external/arcface50.py", line 19, in load_weights
weights_dict = np.load(weight_file).item()
File "/////miniconda3/envs/ostec/lib/python3.6/site-packages/numpy/lib/npyio.py", line 440, in load
pickle_kwargs=pickle_kwargs)
File "/////miniconda3/envs/ostec/lib/python3.6/site-packages/numpy/lib/format.py", line 693, in read_array
array = pickle.load(fp, **pickle_kwargs)
_pickle.UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_ostec.py", line 76, in
main(args)
File "run_ostec.py", line 28, in main
operator = Operator(args)
File "/////OSTeC/core/operator.py", line 112, in init
self.arcface_handler = Arcface_Handler()
File "/////OSTeC/core/arcface_handler.py", line 85, in init
self.emb_norm, _, _, vars = identity_features(aligned_img, 'input_id_features')
File "/////OSTeC/core/arcface_handler.py", line 74, in identity_features
input, embedding, content = arcface50.KitModel('models/fr_models/arcface50.npy', input * 255)
File "/////OSTeC/external/arcface50.py", line 28, in KitModel
__weights_dict = load_weights(weight_file)
File "/////OSTeC/external/arcface50.py", line 21, in load_weights
weights_dict = np.load(weight_file, encoding='bytes').item()
File "/////miniconda3/envs/ostec/lib/python3.6/site-packages/numpy/lib/npyio.py", line 440, in load
pickle_kwargs=pickle_kwargs)
File "/////miniconda3/envs/ostec/lib/python3.6/site-packages/numpy/lib/format.py", line 693, in read_array
array = pickle.load(fp, **pickle_kwargs)
_pickle.UnpicklingError: pickle data was truncated`

Looking for your reply. Thank you very much!

Hi, the shape of im_masked is (3, 1120, 1120), and then concats with the 2d face segmentation mask mask_landmarks. What does the renderer do to the four-channel texture map? It will render RGB and mask channel seperately? What the new mask img_uv_src.pixels[3]<0.5 mean?

Thanks.

conda env create -f environment.yml -n ostec
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:

• dbus==1.13.6=h48d8840_2
• hdf5==1.10.6=nompi_h6a2412b_1114
• menpo3d==0.8.3=py36h355b2fd_0
• gst-plugins-base==1.18.4=hf529b03_2
• libgomp==9.3.0=h5101ec6_17
• ffmpeg==4.3.2=hca11adc_0
• glew==2.1.0=h9c3ff4c_2
• bzip2==1.0.8=h7f98852_4
• libogg==1.3.4=h7f98852_1
• xorg-xextproto==7.3.0=h7f98852_1002
• pyqt-impl==5.12.3=py36h7ec31b9_7
• freetype==2.10.4=h0708190_1
• loguru==0.5.3=py36h5fab9bb_2
• gstreamer==1.18.4=h76c114f_2
• libssh2==1.9.0=ha56f1ee_6
• pyqtchart==5.12=py36h7ec31b9_7
• c-ares==1.17.1=h7f98852_1
• qt==5.12.9=hda022c4_4
• cytoolz==0.11.0=py36h8f6f2f9_3
• scikit-image==0.17.2=py36h284efc9_4
• harfbuzz==2.8.2=h83ec7ef_0
• libstdcxx-ng==9.3.0=hd4cf53a_17
• alsa-lib==1.2.3=h516909a_0
• _openmp_mutex==4.5=1_gnu
• mysql-common==8.0.25=ha770c72_2
• curl==7.78.0=hea6ffbf_0
• libevent==2.1.10=hcdb4288_3
• libdeflate==1.7=h7f98852_5
• openjpeg==2.4.0=hb52868f_1
• xorg-libxt==1.2.1=h7f98852_2
• libvorbis==1.3.7=h9c3ff4c_0
• pthread-stubs==0.4=h36c2ea0_1001
• pyqtwebengine==5.12.1=py36h7ec31b9_7
• certifi==2020.12.5=py36h06a4308_0
• libcblas==3.9.0=11_linux64_openblas
• py-opencv==4.5.2=py36hcb3619a_1
• xorg-libxext==1.3.4=h7f98852_1
• gl2ps==1.4.2=h0708190_0
• libgcc-ng==9.3.0=h5101ec6_17
• moderngl==5.6.4=py36h284efc9_0
• gettext==0.19.8.1=h0b5b191_1005
• libgfortran-ng==11.2.0=h69a702a_11
• libxkbcommon==1.0.3=he3ba5ed_0
• readline==8.1=h27cfd23_0
• fontconfig==2.13.1=hba837de_1005
• xorg-libxau==1.0.9=h7f98852_0
• opencv==4.5.2=py36h5fab9bb_1
• libxcb==1.13=h7f98852_1003
• pillow==8.3.1=py36h676a545_0
• libcurl==7.78.0=h2574ce0_0
• krb5==1.19.2=hcc1bbae_0
• libev==4.33=h516909a_1
• menpo==0.11.0=py36h355b2fd_0
• setuptools==58.0.4=py36h06a4308_0
• graphite2==1.3.13=h58526e2_1001
• libopenblas==0.3.17=pthreads_h8fe5266_1
• xz==5.2.5=h7b6447c_0
• lz4-c==1.9.3=h9c3ff4c_1
• libopus==1.3.1=h7f98852_1
• expat==2.4.1=h9c3ff4c_0
• tornado==6.1=py36h8f6f2f9_1
• xorg-xproto==7.0.31=h7f98852_1007
• traits==6.2.0=py36h8f6f2f9_0
• x264==1!161.3030=h7f98852_1
• liblapacke==3.9.0=11_linux64_openblas
• zstd==1.5.0=ha95c52a_0
• openh264==2.1.1=h780b84a_0
• cyvlfeat==0.7.0=py36h0280710_0
• libnetcdf==4.8.0=nompi_hcd642e3_103
• xorg-libxrender==0.9.10=h7f98852_1003
• libpng==1.6.37=h21135ba_2
• lcms2==2.12=hddcbb42_0
• jpeg==9d=h36c2ea0_0
• libtheora==1.1.1=h7f98852_1005
• xorg-kbproto==1.0.7=h7f98852_1002
• jasper==1.900.1=h07fcdf6_1006
• libglib==2.68.3=h3e27bee_0
• icu==68.1=h58526e2_0
• libblas==3.9.0=11_linux64_openblas
• libpq==13.3=hd57d9b9_0
• libtiff==4.3.0=hf544144_1
• tbb==2020.2=h4bd325d_4
• xorg-libice==1.0.10=h7f98852_0
• gmp==6.2.1=h58526e2_0
• pugixml==1.11.4=h9c3ff4c_0
• cairo==1.16.0=h6cf1ce9_1008
• ca-certificates==2021.9.30=h06a4308_1
• libclang==11.1.0=default_ha53f305_1
• nettle==3.6=he412f7d_0
• openssl==1.1.1l=h7f8727e_0
• double-conversion==3.1.5=h9c3ff4c_2
• proj==7.2.0=h277dcde_2
• libuuid==2.32.1=h7f98852_1000
• yaml==0.2.5=h516909a_0
• libgfortran5==11.2.0=h5c6108e_11
• xorg-renderproto==0.11.1=h7f98852_1002
• glib-tools==2.68.3=h9c3ff4c_0
• immutables==0.15=py36h8f6f2f9_0
• zlib==1.2.11=h7b6447c_3
• glcontext==2.3.4=py36hc4f0c31_0
• tk==8.6.11=h1ccaba5_0
• python==3.6.13=h12debd9_1
• nss==3.67=hb5efdd6_0
• pip==21.0.1=py36h06a4308_0
• pywavelets==1.1.1=py36h92226af_3
• pixman==0.40.0=h36c2ea0_0
• libglu==9.0.0=he1b5a44_1001
• libopencv==4.5.2=py36hb84549a_1
• mysql-libs==8.0.25=hfa10184_2
• tbb-devel==2020.2=h4bd325d_4
• kiwisolver==1.3.1=py36h605e78d_1
• utfcpp==3.2.1=ha770c72_0
• libffi==3.3=he6710b0_2
• scipy==1.5.3=py36h9e8f40b_0
• libxml2==2.9.12=h72842e0_0
• vtk==9.0.1=no_osmesa_py36hfa3a401_109
• libiconv==1.16=h516909a_0
• lame==3.100=h7f98852_1001
• pyqt5-sip==4.19.18=py36hc4f0c31_7
• lerc==2.2.1=h9c3ff4c_0
• jsoncpp==1.9.4=h4bd325d_3
• pyqt==5.12.3=py36h5fab9bb_7
• pathlib==1.0.1=py36h5fab9bb_4
• ncurses==6.2=he6710b0_1
• liblapack==3.9.0=11_linux64_openblas
• nspr==4.30=h9c3ff4c_0
• h5py==2.10.0=py36hd6299e0_1
• importlib-metadata==4.8.1=py36h5fab9bb_0
• xorg-libx11==1.7.2=h7f98852_0
• gnutls==3.6.13=h85f3911_1
• xorg-libsm==1.2.3=hd9c2040_1000
• eigen==3.3.9=h4bd325d_1
• libllvm11==11.1.0=hf817b99_2
• libprotobuf==3.16.0=h780b84a_0
• pcre==8.45=h9c3ff4c_0
• libzip==1.8.0=h4de3113_0
• xorg-libxdmcp==1.1.3=h7f98852_0
• libwebp-base==1.2.0=h7f98852_2
• libedit==3.1.20191231=he28a2e2_2
• libnghttp2==1.43.0=h812cca2_0
• vlfeat==0.9.20=h14c3975_1002
• ld_impl_linux-64==2.35.1=h7274673_9
• glib==2.68.3=h9c3ff4c_0
• imagecodecs-lite==2019.12.3=py36h92226af_3
• jbig==2.1=h7f98852_2003
• sqlite==3.36.0=hc218d9a_0
• hdf4==4.2.15=h10796ff_3

Hi, I think here is a transformation, moving the coordinate system zero point from the center to the lower left corner, but why dense landmarks need to minus 5?

https://github.com/barisgecer/Deep3DFaceRecon_pytorch/blob/e3020cf33a0d32394d32862e6b3779151b361045/ostec_api.py#L88-L89

active conda and all that
when I install everything and proceed to run the script

python run_ostec.py --source_dir [source_dir] --save_dir [save_dir] [-f] -i [iterations (default 200)] -m [soft | hard | auto]

first error

Traceback (most recent call last):
  File "run_ostec.py", line 12, in <module>
    import menpo.io as mio
ModuleNotFoundError: No module named 'menpo'

then! pip install menpo

then another mistake

/usr/local/lib/python3.7/site-packages/menpo/image/base.py:25: UserWarning: Falling back to scipy interpolation for affine warps
  warn("Falling back to scipy interpolation for affine warps")
Traceback (most recent call last):
  File "run_ostec.py", line 17, in <module>
    from core.operator import Operator
  File "/content/OSTeC/core/operator.py", line 7, in <module>
    from core.projection_handler import Projection_Handler
  File "/content/OSTeC/core/projection_handler.py", line 13, in <module>
    import external.stylegan2.dnnlib.tflib as tflib
  File "/content/OSTeC/external/stylegan2/dnnlib/tflib/__init__.py", line 7, in <module>
    from . import autosummary
  File "/content/OSTeC/external/stylegan2/dnnlib/tflib/autosummary.py", line 26, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

which version of tensorflow should I install

It seems that Conda did not install some things so I installed it myself but now it throws this error

/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
/usr/local/lib/python3.7/site-packages/kornia/augmentation/augmentation.py:1875: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead.
  category=DeprecationWarning,
Traceback (most recent call last):
  File "/content/OSTeC/run_ostec.py", line 74, in <module>
    main(args)
  File "/content/OSTeC/run_ostec.py", line 27, in main
    operator = Operator(args)
  File "/content/OSTeC/core/operator.py", line 94, in __init__
    self.tcoords_full = mio.import_pickle('models/topology/tcoords_full.pkl')
  File "/usr/local/lib/python3.7/site-packages/menpo/io/input/base.py", line 484, in import_pickle
    return _import(filepath, pickle_types, importer_kwargs=kwargs)
  File "/usr/local/lib/python3.7/site-packages/menpo/io/input/base.py", line 912, in _import
    raise ValueError("{} is not a file".format(path))
ValueError: /content/OSTeC/external/deep3dfacerecon/models/topology/tcoords_full.pkl is not a file

The result of mine is a little strange, my texture and shape are not aligned, especially at nose eyes and mouth.

I haven't tested it,will it preserve high resolution detail of photos in generated textures (like 5000*5000)?

Hi,
I followed your installation and environment requirements.
I pass all the model loading and compilations successfully, but then in line 95 in file arcface_handler.py:
return self.sess.run(self.emb_norm, {self.img_ph: [img.pixels_with_channels_at_back()], self.lms_ph:[lms]})
it crashes with error:
Check failed: cusolverDnCreate(&cusolver_dn_handle) == CUSOLVER_STATUS_SUCCESS Failed to create cuSolverDN instance
any idea why this happens? (tried to google it, but did not find anything that was not in the code already as config.gpu_options.allow_growth = True
Any idea how I can fix this?
Thanks

As per the subject line; tried to frontalize the photo of Christopher Lee (top left input in Fig.1), the result looks nothing like yours for some reason...

Hi Baris, thanks for sharing your work, I tried to run the code ( in google colab). the FaceRecon Model was created, also the test_opt.txt. but there is no result (images).

Using TensorFlow backend.
2022-02-14 16:52:33.113699: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-02-14 16:52:33.119043: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299995000 Hz
2022-02-14 16:52:33.119314: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5644886c7d40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-02-14 16:52:33.119366: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2022-02-14 16:52:33.121564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2022-02-14 16:52:33.252205: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:33.253213: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56448856bd40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-02-14 16:52:33.253252: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2022-02-14 16:52:33.253528: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:33.254265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
2022-02-14 16:52:33.254699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-02-14 16:52:33.256152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2022-02-14 16:52:33.257373: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2022-02-14 16:52:33.257785: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2022-02-14 16:52:33.259474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2022-02-14 16:52:33.260793: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2022-02-14 16:52:33.264730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-02-14 16:52:33.264851: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:33.265688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:33.266405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2022-02-14 16:52:33.266498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-02-14 16:52:33.268258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-02-14 16:52:33.268321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2022-02-14 16:52:33.268339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2022-02-14 16:52:33.268511: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:33.269298: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:33.270104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10813 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2022-02-14 16:52:35.228822: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:35.229581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
2022-02-14 16:52:35.229682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-02-14 16:52:35.229742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2022-02-14 16:52:35.229792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2022-02-14 16:52:35.229851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2022-02-14 16:52:35.229896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2022-02-14 16:52:35.229942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2022-02-14 16:52:35.229987: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-02-14 16:52:35.230102: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:35.230899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:35.231592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2022-02-14 16:52:35.231650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-02-14 16:52:35.231673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2022-02-14 16:52:35.231689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2022-02-14 16:52:35.231805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:35.232608: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:35.233400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10813 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... 2022-02-14 16:52:36.641169: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:36.641936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
2022-02-14 16:52:36.642055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-02-14 16:52:36.642117: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2022-02-14 16:52:36.642165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2022-02-14 16:52:36.642255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2022-02-14 16:52:36.642314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2022-02-14 16:52:36.642360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2022-02-14 16:52:36.642407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-02-14 16:52:36.642506: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:36.643388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:36.644114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2022-02-14 16:52:36.644167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-02-14 16:52:36.644188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2022-02-14 16:52:36.644202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2022-02-14 16:52:36.644335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:36.645056: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:36.645756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 10813 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
Loading... Done.
Setting up TensorFlow plugin "upfirdn_2d.cu": Preprocessing... 2022-02-14 16:52:37.640597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:37.641460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
2022-02-14 16:52:37.641571: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-02-14 16:52:37.641625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2022-02-14 16:52:37.641674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2022-02-14 16:52:37.641726: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2022-02-14 16:52:37.641774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2022-02-14 16:52:37.641821: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2022-02-14 16:52:37.641871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-02-14 16:52:37.641986: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:37.642752: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:37.643515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2022-02-14 16:52:37.643576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-02-14 16:52:37.643600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2022-02-14 16:52:37.643616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2022-02-14 16:52:37.643758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:37.644520: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-14 16:52:37.645297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 10813 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
Loading... Done.
2022-02-14 16:52:51.391704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2022-02-14 16:52:51.524127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
Loading ResNet Model:
2022-02-14 16:54:07.694218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-02-14 16:54:07.694277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]
MTCNN loaded
model [FaceReconModel] was created
loading the model from /content/drive/MyDrive/OSTeC1/external/deep3dfacerecon/checkpoints/face_recon/epoch_latest.pth
Scanning paths...
Scanning paths...

Hello, I tried to capture the trimesh generated by Operator#run:

face = self.get_tmesh(im, reconstruction_dict)
results_dict['face'] = face

and then export it using menpo3d io:

import menpo3d.io as m3dio
...
m3dio.export_textured_mesh(results_dict["face"].tmesh, save_path.replace(ext, ".obj"), extension="obj", overwrite=True)

but the generated mesh is corrupt:

any idea why? is there a proper way to do so? tried using original code from Deep3DFaceRecon_pytorch and the mesh is ok but it fails to generate texture due tensor size missmatch.

First of all, thank you very much for sharing such excellent work. I want to put the last generated texture into meshlab for analysis, so I generated an obj file, which contains 3D face vertices and corresponding faces, but there was a problem when adding 2D texture vertices and faces. The complete 2D faces index is not found. Can you tell me how to get the complete 2D texture vertices and faces? Thank you again! Looking forward to your reply!

Hi, Baris!
I love your job, wonderful job!! However, there is a question about how to align the S′_i and M_i to aligned rendering picture I_i.
It's obvious that I_i can be aligned by converting it to PIL image and feeding into the ffhq dataset face alignment pipeline. But there is no clue about how to align the projected face model S'_i to the aligned rendering picture to make S'_i to be landmark correspondent to I_i.
I would be very appreciate if you can answer my question. Thank you again for such amazing job!!
Best regards,
Rodney.

Hello, can anyone please share their working conda environment + cuda and cudnn + python versions. I've installed everything according to the instructions and also installed the environment with the provided yml but still cant run the code.

Hello, I am very interested in your work. What if I wanted to train this network from scratch? Can you give me detailed training procedures?

Can you help me please to solve this problem? I try to run test.py for Deep3Dfacerecon.

----------------- Options ---------------
add_image: True
bfm_folder: BFM
bfm_model: BFM_model_front.mat
camera_d: 10.0
center: 112.0
checkpoints_dir: ./checkpoints
dataset_mode: None
ddp_port: 12355
display_per_batch: True
epoch: latest
eval_batch_nums: inf
focal: 1015.0
gpu_ids: 0
img_folder: examples
init_path: checkpoints/init_model/resnet50-0676ba61.pth
isTrain: False [default: None]
model: facerecon
name: face_recon
net_recon: resnet50
phase: test
suffix:
use_ddp: False [default: True]
use_last_fc: False
verbose: False
vis_batch_nums: 1
world_size: 1
z_far: 15.0
z_near: 5.0
----------------- End -------------------
model [FaceReconModel] was created
loading the model from ./checkpoints/face_recon/epoch_latest.pth
0 examples/000002.jpg
Traceback (most recent call last):
File "test.py", line 72, in
main(0, opt,opt.img_folder)
File "test.py", line 62, in main
model.test() # run inference
File "/content/drive/MyDrive/Deep3DFaceRecon_pytorch-baris/models/base_model.py", line 162, in test
self.forward()
File "/content/drive/MyDrive/Deep3DFaceRecon_pytorch-baris/models/facerecon_model.py", line 137, in forward
self.facemodel.compute_for_render(output_coeff)
File "/content/drive/MyDrive/Deep3DFaceRecon_pytorch-baris/models/bfm.py", line 303, in compute_for_render
face_color = self.compute_color(face_texture, face_norm_roted, coef_dict['gamma'])
File "/content/drive/MyDrive/Deep3DFaceRecon_pytorch-baris/models/bfm.py", line 176, in compute_color
face_color = torch.cat([r, g, b], dim=-1) * face_texture
RuntimeError: The size of tensor a (35709) must match the size of tensor b (53215) at non-singleton dimension 1