linsen20220222 / clip-mesh-smplx

CLIP + Mesh + SMPL-X

CLIP-Mesh-SMPLX

The following was a research project initially inspired by CLIPMatrix. However, after the release of CLIP-Actor and AvatarCLIP I've decided to just release my code since those works have extensively documented this domain.

The main difference is that this uses meshes which are much faster and less memory-intensive, additionally I explore multiple independent optimizable parameters

IMO there is still huge for improvement on this topic and hopefully this repo allows everyone to perform quick experimentation without needing large compute - Enjoy! 🎉

License

This project is for research purposes only

SMPL-X

This codebase uses SMPL-X Models and the smplx pip library. By using this code you agree to the SMPL-X Model License and smplx License

Renderer

This code base relies on nvdiffmodeling and in turn nvdiffrast.

Demos

Get started by testing out the features through the collabs

Create a character from just a text prompt	Change an expression with text prompt	Pose with a description (⚠️ WIP)	Create fantastic creatures like CLIPMatrix

Setup

git clone --recurse-submodules [email protected]:NasirKhalid24/CLIPMesh-SMPLX.git
cd CLIPMesh-SMPLX

# or manually unzip so the .npz files are in the folder as shown below
unzip /models/smplx/smplx_npz.zip -d /models/smplx   

pip install virtualenv
virtualenv ENV
source ENV/bin/activate

pip install --upgrade pip
pip install -r requirements.txt

CLIPMesh-SMPLX
│
└───models
│   └───smplx
│       │   SMPLX_NEUTRAL.npz
│       │   ....
│       │   ....

To replicate the demos you can use the configs provided.

# For a single character generation
python main.py --path=configs/single.yaml

# For a single expression generation
python main.py --path=configs/expression.yaml

# For a single pose generation
python main.py --path=configs/pose.yaml

# For a CLIPMatrix type generation
python main.py --path=configs/pose.yaml

# For something else use params
python main.py \
--optim   body expression texture normal \
--options face full back \
--epochs  1000 \
--gpu     0 \
--face_text "a thin 3D render of the face of a James Bond" \
--full_text "a thin 3D render of the James Bond" \
--back_text "a thin 3D render of the back of James Bond" \
--debug_log true
--log_int 250