songrise / avatarcraft Goto Github PK

In the paper, the total loss is the sum of sds loss,sil loss and eikonal loss. But I find a new loss named stylization loss in released code,I don't see what this loss does. Could you please explain the stylization loss?

Hey, really happy to see that AvatarCraft is finally released.

I observed most of the demos are present as the T-pose. Meanwhile, there are some A-pose demonstrations in Fig.6 of the paper. I wonder how to render an A-pose image of a given model. Do I need to prepare a corresponding motion pkl? Or using some SMPL Parameters would just work?

Hope for your reply :-)

Thank you for your amazing work. Just want to ask when you plan to release your code👀

I would like to know how many hours are needed to train AvatarCraft on a RTX-3090(24G) GPU or how many hours are needed to train AvatarCraft on a A100(40G) GPU?

Thanks for great work!
To render articulated avatar, I tried yoga pose in AMASS SFU dataset.
Then encountered an error:

Traceback (most recent call last):
  File "render_warp.py", line 272, in <module>
    main(opt)
  File "render_warp.py", line 50, in main
    max_frames=opt.max_frames
  File "render_warp.py", line 176, in calc_local_trans
    concat_joints=True
  File "/workspace/jaeseok/avatarcraft/models/smpl.py", line 152, in verts_transformations
    return_T=True, concat_joints=concat_joints)
  File "/workspace/jaeseok/avatarcraft/models/smpl.py", line 396, in lbs
    v_delta = blend_shapes(betas, shapedirs)
  File "/workspace/jaeseok/avatarcraft/models/smpl.py", line 546, in blend_shapes
    blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
  File "/opt/conda/envs/craft/lib/python3.7/site-packages/torch/functional.py", line 406, in einsum
    return einsum(equation, *_operands)
  File "/opt/conda/envs/craft/lib/python3.7/site-packages/torch/functional.py", line 408, in einsum
    return _VF.einsum(equation, operands)  # type: ignore
einsum() operands do not broadcast with remapped shapes [original->remapped]: [1, 10]->[1, 1, 1, 10] [6890, 3, 300]->[1, 6890, 3, 300]

Great work!!! May I ask when the code will be released?

First of all, thank you for excellent work!
Anyway, I tried to reconstruct bare_smpl model using reconstruct.py and da_smpl_512 dataset in your repo.
But naively executing the code using the dataset, reconstruction quality is really bad.

Would you let me know how to train it?

Your work AvatarCraft impressed me a lot and it inspired me a lot.

But when I use the following command to create an avatar with text prompt, the results are far from expectation.
python stylize.py --weights_path "ckpts/bare_smpl.pth.tar" --tgt_text "Hulk, photorealistic style" --exp_name "hulk" --batch_size 4096

Here shows the canonical render result:

Are these results normal?
Also, we found there seemed to be a mode collapse after 4800iter. Can you help with this problem?

Ps. We test this code on Win10+cuda 11.1 and RTX3090-24G, and we download the stable-diffusion from huggingface instead of importing it online.

First of all, thank you for excellent work!
But,I meet this problem when I run python render_canonical.py to render the canonical avatar . Is there any solution?

Traceback (most recent call last):
File "/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/miniconda3/envs/avatar/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "render_canonical.py", line 30, in
import models.instant_nsr as instant_nsr
File "/AvatarCraft-main/models/instant_nsr.py", line 19, in
from encoder import get_encoder
File "/AvatarCraft-main/encoder/init.py", line 2, in
from encoder.hashencoder import HashEncoder
File "/AvatarCraft-main/encoder/hashencoder/init.py", line 1, in
from .hashgrid import HashEncoder
File "/AvatarCraft-main/encoder/hashencoder/hashgrid.py", line 9, in
from .backend import backend
File "/AvatarCraft-main/encoder/hashencoder/backend.py", line 12, in
sources=[os.path.join(src_path, 'src', f) for f in [
File "/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in jit_compile
is_standalone=is_standalone)
File "/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1407, in write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'hash_encoder': [1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output hashencoder.cuda.o.d -DTORCH_EXTENSION_NAME=hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include -isystem
/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include/TH -isystem
/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include/THC -isystem
/miniconda3/envs/avatar/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -U__CUDA_NO_HALF_OPERATORS -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /AvatarCraft-main/encoder/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o
FAILED: hashencoder.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output hashencoder.cuda.o.d -DTORCH_EXTENSION_NAME=hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem
/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include -isystem
/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem
/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include/TH -isystem
/miniconda3/envs/avatar/lib/python3.7/site-packages/torch/include/THC -isystem
/miniconda3/envs/avatar/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /AvatarCraft-main/encoder/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
ninja: build stopped: subcommand failed.

This is the error:

Env:
CUDA11.3+pytorch1.12.0+pytorch3D 0.7.4+nvidia cub 1.11.0

Both paper button and arxiv button in the project page here is linked with Nerfie. I wonder whether it is a mistake or the paper is just not available for reading yet. If the situation is the latter one, then when will the paper be released? Hope for your answer : )

BTW, it is a REALLY great work and demo. Your team's work is always superising me

Amazing work, will the code be released soon?

Hi, thanks for the great work.

However, I cannot get the articulation code to work, as there seem to be some missing files, including "data/stand_pose.npy", "data/smpl_da_512" and 'data/smplx/smpl_uv.obj'.

Are there any more updates planned?
Thanks.

python stylize.py --weights_path "ckpts/bare_smpl.pth.tar" --tgt_text "Hulk, photorealistic style" --exp_name "hulk" --batch_size 4096
No CUDA runtime is found, using CUDA_HOME=':/usr/local/cuda-11.3'
Traceback (most recent call last):
  File "stylize.py", line 17, in <module>
    from models import instant_nsr, diffusion
  File "/data2/gaoyufei_m24/AvatarCraft/models/instant_nsr.py", line 19, in <module>
    from encoder import get_encoder
  File "/data2/gaoyufei_m24/AvatarCraft/encoder/__init__.py", line 2, in <module>
    from encoder.hashencoder import HashEncoder
  File "/data2/gaoyufei_m24/AvatarCraft/encoder/hashencoder/__init__.py", line 1, in <module>
    from .hashgrid import HashEncoder
  File "/data2/gaoyufei_m24/AvatarCraft/encoder/hashencoder/hashgrid.py", line 9, in <module>
    from .backend import _backend
  File "/data2/gaoyufei_m24/AvatarCraft/encoder/hashencoder/backend.py", line 12, in <module>
    sources=[os.path.join(_src_path, 'src', f) for f in [
  File "/data2/gaoyufei_m24/mambaforge/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
    keep_intermediates=keep_intermediates)
  File "/data2/gaoyufei_m24/mambaforge/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in _jit_compile
    is_standalone=is_standalone)
  File "/data2/gaoyufei_m24/mambaforge/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1400, in _write_ninja_file_and_build_library
    is_standalone=is_standalone)
  File "/data2/gaoyufei_m24/mambaforge/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1782, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "/data2/gaoyufei_m24/mambaforge/envs/avatar/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range

envs

NVIDIA 3090 24G

NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

reproduce

clone the repo
enviroment setup with cuda version 11.x
data setup without optional
run python stylize.py --weights_path "ckpts/bare_smpl.pth.tar" --tgt_text "Hulk, photorealistic style" --exp_name "hulk" --batch_size 4096

Hi, excellent work! And when will u pls release the code?