sunwood-ai-labs / sifu

[CVPR 2024] Official repository for paper "SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction"

SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction

Zechuan Zhang  Zongxin Yang  Yi Yang✉ 

ReLER, CCAI, Zhejiang University

^✉Corresponding Author

CVPR 2024

Figure 1. With just a single image, SIFU is capable of reconstructing a high-quality 3D clothed human model, making it well-suited for practical applications such as 3D printing and scene creation. At the heart of SIFU is a novel Side-view Conditioned Implicit Function, which is key to enhancing feature extraction and geometric precision. Furthermore, SIFU introduces a 3D Consistent Texture Refinement process, greatly improving texture quality and facilitating texture editing with the help of text-to-image diffusion models. Notably proficient in dealing with complex poses and loose clothing, SIFU stands out as an ideal solution for real-world applications.

📖 For more visual results, go checkout our project page

This repository will contain the official implementation of SIFU.

News

• [2024/2/28] We release the code of geometry reconstruction, including test and inference.
• [2024/2/27] SIFU has been accepted by CVPR 2024! See you in Seattle!
• [2023/12/13] We release the paper on arXiv.
• [2023/12/10] We build the Project Page.

Installation

• Ubuntu 20 / 18
• CUDA=11.6, GPU Memory > 16GB
• Python = 3.8
• PyTorch = 1.13.0 (official Get Started)
• PyTorch3D (official INSTALL.md, recommend install-from-local-clone)

git clone https://github.com/River-Zhang/SIFU.git
sudo apt-get install libeigen3-dev ffmpeg
cd SIFU
conda env create -f environment.yaml
conda activate sifu
pip install -r requirements.txt

Please download the checkpoint (one drive) and place them in ./data/ckpt

Please follow ICON to download the extra data, such as HPS and SMPL.

Inference

python -m apps.infer -cfg ./configs/sifu.yaml -gpu 0 -in_dir ./examples -out_dir ./results -loop_smpl 100 -loop_cloth 200 -hps_type pixie

Testing

# 1. Register at http://icon.is.tue.mpg.de/ or https://cape.is.tue.mpg.de/
# 2. Download CAPE testset (Easy: 50, Hard: 100)
bash fetch_cape.sh 

# evaluation
python -m apps.train -cfg ./configs/train/sifu.yaml -test

# TIP: the default "mcube_res" is 256 in apps/train.

Applications of SIFU

Scene Building

3D Printing

Texture Editing

Animation

In-the-wild Reconstruction