abdallahdib / nextface Goto Github PK

Hello,
I have this problem when i run the code, and I want to ask for your help.

detecting landmarks...
init camera pose...
1/3 => Optimizing head pose and expressions using landmarks...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:41<00:00, 47.85it/s]
2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light...
0%| | 0/401 [00:00<?, ?
0%|▎ | 1/401 [00:29<3:16:12,
terminate called after throwing an instance of 'thrust::system::system_error'
what(): device free failed: an illegal memory access was encountered
run.sh: line 4: 14367 Aborted (core dumped) python optimizer.py --sharedIdentity --input /image

I run the code with GeForce RTX 2080 Ti GPU, and python==3.6.7, torch==1.9.1, cuda toolkit==10.2.
Thanks a lot.

Hi, thanks for sharing your work!

I've come across a couple of problems,
Firstly, I'm trying to reproduce on python 3.9 and as you may know redner doesn't have a build for that version yet. I've worked around the problem by removing the imports and functions that call the library (super dirty, I know).

But now I'm facing this issue in step 2 of 3 and can't tell where the problem originates

2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light... 0%| | 0/401 [00:00<?, ?it/s]C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [64,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [65,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [66,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [70,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [71,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [72,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [28,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [29,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [5563,0,0], thread: [30,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. 0%| | 0/401 [00:00<?, ?it/s] Traceback (most recent call last): File "B:\NextFace\optimizer.py", line 468, in <module> optimizer.run(inputDir, File "B:\NextFace\optimizer.py", line 425, in run self.runStep2() File "B:\NextFace\optimizer.py", line 233, in runStep2 images = self.pipeline.render(cameraVerts, diffuseTextures, specularTextures) File "B:\NextFace\pipeline.py", line 114, in render envMaps = self.sh.toEnvMap(self.vShCoeffs) File "B:\NextFace\sphericalharmonics.py", line 81, in toEnvMap envMaps = torch.zeros( [shCoeffs.shape[0], self.resolution[0], self.resolution[1], 3]).to(shCoeffs.device) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I'm on Windows 10, using an Nvidia RTX2070s and tried with texture size 512, 1024 and 2048 all with the same result.
I've read on another issue with similar error messages where the issue was that the wrong basel model was being used, but I've triple checked and now I not sure where to check. Could it be because of the redner library? That seems unlikely to me

Cheers!

PS The text is wonky because of the '' so here's an image

I installed everything correctly using the INSTALL file and when testing it, step 1 works fine, but then it gets stuck on step 2.
Says 0/401 and unknown iterations per second.
how long should I wait to see if it does eventually end?
I'm on a 3060ti. (And I tried with cpu, does the same thing)
thanks!

2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light...
0%| | 0/401 [00:00<?, ?it/s]python3: /tmp/pip-req-build-4hdcr8rl/src/scene.cpp:125: Scene::Scene(const Camera&, const std::vector<const Shape*>&, const std::vector<const Material*>&, const std::vector<const AreaLight*>&, const std::shared_ptr&, bool, int, bool, bool): Assertion `false' failed.
Aborted (core dumped)

core dumped when execute step2. Is anyone have confused by this issue?

Hi, thank you for nice work.
It's great research to make high resolution face texture.
In my case, result set(diffuseMap, roughnessMap, specularMap, envMap) is too difficult to use (compare than other project. they make .obj and texture image only)
I know these are more detail features. But I'm using pytorch3d to visualize 3D obj, and there is no option to use these.
Is there any solution to merge these details using python?
I want to get reconstructed .obj result without 3D tools (unreal, Unity, blender, etc)

I can't start the GPU 3060

Cant seem to find a solution to this and its getting pretty late :')
I tried adding os.environ['CUDA_LAUNCH_BLOCKING'] = "1" but it didnt help, got any ideas? thanks a head of time :))

Edit:
I added os.environ['CUDA_LAUNCH_BLOCKING'] = "1 to the classes optimizer and morphablemodel, no diff

Hello, thanks for your wonderful job! I have tried to render the image with ray tracing, but it seems to have lots of noise. Do you have any idea to denoise the images? Thank you again!

How to convert the vertex into FLAME mode?

After exporting mesh and texture maps,can we edit the texture and run the program with the edited texture fixed?

Hi abdallah,

NextFace result is amazing, while I meet some problems in the brdf formulation.
In paper, you say the roughness is static, and I see u put it as 0.4

May I know, why do this or why this work?

Best
Jiaxiang

python optimizer.py --input /home/dingwuyou/Downloads/83aedae4baa21cd2868437d5ceee8e22.jpeg --output persion1
loading optim config from: ./optimConfig.ini Loading Basel Face Model 2017 from ./baselMorphableModel/morphableModel-2017.pickle...
loading mesh normals... loading uv parametrization...
loading landmarks association file... creating sampler...
loading image from path: /home/dingwuyou/Downloads/83aedae4baa21cd2868437d5ceee8e22.jpeg [INFO] resizing input image to fit: 256 px resolution...
detecting landmarks... init camera pose...
1/3 => Optimizing head pose and expressions using landmarks... 100%|█████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:04<00:00, 431.23it/s]
2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light... 0%| | 0/401 [00:00<?, ?it/s]256
torch.Size([1, 512, 512, 3]) /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1440,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" fail
ed. /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1444,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" fail
ed. /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1444,0,0], thread: [33,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" fail
ed. /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" fail
ed.
...
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [77,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [78,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [79,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [86,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [87,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [88,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1502,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
0%| | 0/401 [00:00<?, ?it/s]
Traceback (most recent call last):
File "optimizer.py", line 467, in
doStep3= doStep3)
File "optimizer.py", line 419, in run
self.runStep2()
File "optimizer.py", line 231, in runStep2
specularTextures = self.pipeline.morphableModel.generateTextureFromAlbedo(specAlbedo)
File "/home/dingwuyou/Projects/NextFace/morphablemodel.py", line 169, in generateTextureFromAlbedo
neighboors = torch.arange(self.faces.shape[-1], dtype = torch.int64, device = self.faces.device)
RuntimeError: tabulate: failed to synchronize: cudaErrorAssert: device-side assert triggered

Hi does anyone know if uvParametrization.512.pickle is available at 1024 or higher. If not how to generate this file?

Thanks.

Hi, thanks for your great work!
I've run the following command line for s3.png and s4.png, and got some weired results. Can you tell me how to get the the same results as reported in your paper? Many thanks!

python optimizer.py --input ./input/s3.png --output ./output
loading optim config from: ./optimConfig.ini
Loading Basel Face Model 2017 from ./baselMorphableModel/morphableModel-2017.pickle...
loading mesh normals...
loading uv parametrization...
loading landmarks association file...
creating sampler...
loading image from path: ./input/s3.png
detecting landmarks...
init camera pose...
1/3 => Optimizing head pose and expressions using landmarks...
100%|█████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:06<00:00, 301.50it/s]
2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light...
100%|████████████████████████████████████████████████████████████████████████████████| 401/401 [02:28<00:00, 2.69it/s]
3/3 => finetuning albedos, shape, expression, head pose and scene light...
100%|████████████████████████████████████████████████████████████████████████████████| 101/101 [00:39<00:00, 2.59it/s]
took 3.25 minutes to optimize
saving to: ' ./output/s3.png/ '. hold on...
diffuseAlbedo.shape[0] = 1

And here are my results. As you can see, both subjects have Asian faces, the bridge of the nose is not so high, but the results are all European and American faces.

Hi, I have tried many images, but the diffuse map around the eye is not fit with the 3D model and was blurry. Could you explaine it? Thanks very much.

Thank you for your great job.

Can we use the output files to make a faceswap video?
We anticipate that we will need a technique to detect faces from a video or images that shows the target person and apply a mask processed from an obj file.

Please forgive me for asking this question without knowing how to handle obj files.

Dear all,
NextFace complained about cannot find any face in the image. I tried to save the image as jpg or png but it didn't work. I even tried to
use your image in Github but without any luck. Sorry it looks like a stupid question but I just don't know how to solve it.

(faceNext) C:\Users\kenmax\NextFace>python optimizer.py --input c:\Users\kenmax --output c:\Users\kenmax\hikari
loading optim config from: ./optimConfig.ini
Loading Basel Face Model 2017 from ./baselMorphableModel/morphableModel-2017.pickle...
loading mesh normals...
loading uv parametrization...
loading landmarks association file...
creating sampler...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
loading images from path: c:\Users\kenmax
loading image from path: c:\Users\kenmax/hikari03.png
[INFO] resizing input image to fit: 256 px resolution...
detecting landmarks using: mediapipe
Traceback (most recent call last):
File "optimizer.py", line 491, in
doStep3= doStep3)
File "optimizer.py", line 420, in run
self.setImage(imagePathOrDir, sharedIdentity)
File "optimizer.py", line 128, in setImage
landmarks = self.landmarksDetector.detect(self.inputImage.tensor)
File "C:\Users\kenmax\NextFace\landmarksmediapipe.py", line 51, in detect
land = self._detect((images[i].detach().cpu().numpy() * 255.0).astype('uint8'))
File "C:\Users\kenmax\NextFace\landmarksmediapipe.py", line 71, in _detect
raise RuntimeError('No face was found in this image')
RuntimeError: No face was found in this image

(faceNext) C:\Users\kenmax\NextFace>

Little help, please. I am getting below error.

(nextface) E:\GITHUB\NEXTFACE\NextFace>python optimizer.py --sharedIdentity --input input/s2.png --output output/
loading optim config from: ./optimConfig.ini
[WARN] no cuda enabled device found. switching to cpu...
Loading Basel Face Model 2017 from ./baselMorphableModel/morphableModel-2017.pickle...
loading mesh normals...
loading uv parametrization...
loading landmarks association file...
creating sampler...
C:\Anaconda\envs\nextface\lib\site-packages\torch\functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\b\abs_f0dma8qm3d\croot\pytorch_1669187301762\work\aten\src\ATen\native\TensorShape.cpp:2895.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
loading image from path: input/s2.png
detecting landmarks using: mediapipe
E:\GITHUB\NEXTFACE\NextFace\landmarksmediapipe.py:55: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at C:\b\abs_f0dma8qm3d\croot\pytorch_1669187301762\work\torch\csrc\utils\tensor_new.cpp:204.)
return torch.tensor(landmarks, device = self.device)
init camera pose...
1/3 => Optimizing head pose and expressions using landmarks...
100%|██████████████████████████████████████████████████████████████████████████████████| 2000/2000 [01:36<00:00, 20.71it/s]
2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light...
0%| | 0/401 [00:00<?, ?it/s]
Traceback (most recent call last):
File "optimizer.py", line 486, in
optimizer.run(inputDir,
File "optimizer.py", line 434, in run
self.runStep2()
File "optimizer.py", line 242, in runStep2
images = self.pipeline.render(cameraVerts, diffuseTextures, specularTextures)
File "E:\GITHUB\NEXTFACE\NextFace\pipeline.py", line 134, in render
scenes = self.renderer.buildScenes(cameraVerts, self.faces32, normals, self.uvMap, diffuseTextures,
File "E:\GITHUB\NEXTFACE\NextFace\renderer.py", line 126, in buildScenes
cam = self.setupCamera(focal[i], self.screenWidth, self.screenHeight)
File "E:\GITHUB\NEXTFACE\NextFace\renderer.py", line 81, in setupCamera
cam = pyredner.Camera(
File "C:\Anaconda\envs\nextface\lib\site-packages\pyredner\camera.py", line 118, in init
self.intrinsic_mat_inv = torch.inverse(self.intrinsic_mat).contiguous()
RuntimeError: inverse: LAPACK library not found in compilation

Any plans for releasing S2F2?

Dear brother Abdallahdib
First of all, your code is really helpful
I face this issue "No module named 'pyredner'" , I am run the code on colab
could you please advice on this issue

In the second step, it will occur this problem? How can I solve it? My torch version is 1.3.1 and CUDA version is 10.1.

Like mitsuba 3, it can be differential renderer and it includes Disney bsdf(which is used by many rendering software).

instead of diffuseMap I want to get the original texture but mapped to uv map , is there any command for this.

Hi, would you be interested in adding NextFace to Hugging Face? The Hub offers free hosting, and it would make your work more accessible and visible to the rest of the ML community. Models/datasets/spaces(web demos) can be added to a user account or organization similar to github.

Example from other organizations:
Keras: https://huggingface.co/keras-io
Microsoft: https://huggingface.co/microsoft
Facebook: https://huggingface.co/facebook

Example spaces with repos:
github: https://github.com/salesforce/BLIP
Spaces: https://huggingface.co/spaces/salesforce/BLIP

github: https://github.com/facebookresearch/omnivore
Spaces: https://huggingface.co/spaces/akhaliq/omnivore

and here are guides for adding spaces/models/datasets to your org

How to add a Space: https://huggingface.co/blog/gradio-spaces
how to add models: https://huggingface.co/docs/hub/adding-a-model
uploading a dataset: https://huggingface.co/docs/datasets/upload_dataset.html

Please let us know if you would be interested and if you have any questions, we can also help with the technical implementation.

I notice that render and debug output all have resolution 512, does that mean per-pixel basis is based on this 512 image ? I just tried 2048 uv and feel that the output is blurry. Should I increase maxRes in image.py ? I have changed all 256/512 to 2048, not sure how will that play out.

Hi Abdallah,

Thanks for open-sourcing your amazing work. I am currently trying to replicate results from the "Practical Face Reconstruction via Differentiable Ray Tracing". The teaser image on Github (I get the same result when I run the code) is different than the teaser from the paper. In particular, there is a lot of shading baked into the albedo map and possibly poor separation between the diffuse and specular/roughness maps. Are there any settings that can be changed to get closer to the results from the paper?

Hi, thanks for your great work!
One question. I was testing the code. I've noticed if you just finished a single image, and want to test another one, it froze when step into the "detecting landmarks...", and it will stay there for ever. Have to kill the console and restart it for the second image. I am using Win10. Any suggestions? Thanks!!!

2/3 => Optimizing shape, statistical albedos, expression, head pose and scene light...
0%| | 0/401 [00:00<?, ?it/s]
Traceback (most recent call last):
File "optimizer.py", line 462, in
optimizer.run(inputDir,
File "optimizer.py", line 419, in run
self.runStep2()
File "optimizer.py", line 233, in runStep2
images = self.pipeline.render(cameraVerts, diffuseTextures, specularTextures)
File "D:\NextFace\pipeline.py", line 124, in render
scenes = self.renderer.buildScenes(cameraVerts, self.faces32, normals, self.uvMap, diffuseTextures,
File "D:\NextFace\renderer.py", line 127, in buildScenes
scene = pyredner.Scene(cam, materials=[mat], objects=[obj], envmap=pyredner.EnvironmentMap(envMap[i]))
File "D:\anaconda3\envs\faceNext\lib\site-packages\pyredner\scene.py", line 56, in init
shape = pyredner.Shape(vertices = obj.vertices,
File "D:\anaconda3\envs\faceNext\lib\site-packages\pyredner\shape.py", line 371, in init
assert(indices.is_contiguous())
AssertionError
Python 3.8 cuda 11.1

Hey there, thanks for your impressive work!
I tried Nextface first and now implemented DeepNextFace by myself, using default resnet152 in pytorch(pretrained on ImageNet), but during step 1, landmark loss decreased in a quite slow way(landmark loss is the same as NextFace), ended up with about 3000.
I think it may related to slow fitting of focal and camera position params. Dosen't find more details about the training strategy in your paper including appendix, can you please help?

Hi,
Thank you for your effort. I am trying to import the obj file into Blender, but I cant see the model rendered, altough I see the mesh file imported.
Have you tried it on Blender? or is it expected to work on Blender?
Thank you again

Hi, I want to know how to get vertex position error and normal error in your paper. Thank u.

Like same person under very different light environment.

Say I have constructed a model of TomCruise, now I want using my face's expression to drive it, how to do that/.

iterStep1 2000 left Deep3DFaceRecon_pytorch, right NextFace.



iterStep1 10000

it would be great if you could setup a colab notebook with upload option.
Such that we could download the 3d render & texture and import it to tools like blender.

Hi, Thanks for your great work!
After reading your paper, I've noticed that you can generate the colored face with the original texture, instead of the one shown in diffuseMap_0.png. How to get it? Thanks!

Hi, I've downloaded all the files for the BFM 2017 model as well as the albedos .h5 file. When I try loading the model it gives me an error message that I am missing 'normals.pickle'. Where can I find this file? There seems to be no mention of it anywhere in the repo. Any help is much appreciated!

Dear author,
I'm new to CG，can you please teach me the way of calculating uvParams? and each params' meaning, like 'uvFaces', 'uvMapFaces', 'uvValidUVMap', 'uvXYMap'and 'uvVertices'.
thanks a lot!

Hi!

Thanks for the great work.
I try to generate side view image from reconstructed 3d face. How can I do that?
Thank you!

Image of the error

Error occurs on the optimizer creation step, it says to "Download "albedoModel2020_face12_albedoPart.h5", and put it inside /content/NextFace/baselMorphableModel/ and run again." But I've already done that as shown in the image.

hey i tested it, its ok but not much better than other 3d avatar apps out there that can also take sideviews.
Is it planned to use additional sideviews for mesh generation? from what i see ,the same male mesh is used on all models and the mesh itself is just slightly adapted, not that much, so in the end its almost like slapping photo on base mesh without artifacts. sideview is the same with all people.
It also takes awhile on 1080ti , would be nice if it could retain the colours of original image and produce material texture of 1024, let it compute mesh on256 but texture should be higher res
also theres an issue with landmarks, when face is wider than base mesh then landmarks places the dots on cheek, not on the border of the face... maybe cause its not perfect front view

Hi ! Thanks for sharing your code.
I've tried the Colab demo file that you uploaded. Then I get mesh0.obj and material0.mtl.
When I tried the obj file, I was able to get the result that only mesh was expressed.
How do you get the results in a form that expresses the color of the character, not just the mesh? I think I make the result by combining obj file and mtl file, but I don't know what to do.
I need your help! Please understand that English may not be smooth using a translator.

'https' is not recognized I am see these message
can u help with these to solve

I nerver know a lib called pyredner, just pyrender
if it is typo, how to able to run your code? if it is a typo, how do u make twice of them?

very very strange...

Hi, abdallahdib,

1. I run your code, the visualization of envMaps is black,
   

2. the visualization of
   renderAll = torch.cat([ torch.cat([inputTensor[i], torch.ones_like(images[i])[..., 3:]], dim = -1), torch.cat([overlay.to(self.device), torch.ones_like(images[i])[..., 3:]], dim = -1), images[i], #illum[i], diffuseAlbedo[self.getTextureIndex(i)], specularAlbedo[self.getTextureIndex(i)], roughnessAlbedo[self.getTextureIndex(i)]], dim=1)
   is transparent as below.
   

The loss figure is here,

Do you have further suggestions, the overlay rendering results seems right.

Thanks for your great works. I want to know the process of calculating uvParametrization.pickle, in order to replace BFM model to FLAME.
Looking forward to your detailed answer.

hi there, its me again.
by using generateTextureFromAlbedo in morphablemodel.py can convert color vertices to uv texures, how can i inverse this process to convert textures to color vertices?

thanks!

Hi! Thanks for your code. I have some questions about smoothing SH coeffs. I notice that during optimization there is no smoothSH operation because the default value of smooth is set to False in the toEnvMap function.

But after optimization, the SH can be smoothed as specified by smoothSH operation in optimConfig.ini:

What about the results if smooth SH coeffs during optimization?