This project is the official implement of Collaborative Neural Rendering using Anime Character Sheets, which aims to genarate vivid dancing videos from hand-drawn anime character sheets(ACS). Watch more demos in our HomePage.
Contributors: @transpchan, @P2Oileen, @hzwer
- NVIDIA GPU + CUDA + CUDNN
- Python 3.6
- Clone this repository
git clone https://github.com/megvii-research/CoNR
- Dependencies
To install all the dependencies, please run the following commands.
cd CoNR
pip install -r requirements.txt
- Download Weights Download weights from Google Drive. Alternatively, you can download from Baidu Netdisk (password:RDxc).
mkdir weights && cd weights
gdown https://drive.google.com/uc?id=1M1LEpx70tJ72AIV2TQKr6NE_7mJ7tLYx
gdown https://drive.google.com/uc?id=1YvZy3NHkJ6gC3pq_j8agcbEJymHCwJy0
gdown https://drive.google.com/uc?id=1AOWZxBvTo9nUf2_9Y7Xe27ZFQuPrnx9i
gdown https://drive.google.com/uc?id=19jM1-GcqgGoE1bjmQycQw_vqD9C5e-Jm
We prepared two Ultra-Dense Pose sequences for two characters. You can generate more UDPs via 3D models and motions refers to our paper. Baidu Netdisk (password:RDxc)
# for short hair girl
gdown https://drive.google.com/uc?id=11HMSaEkN__QiAZSnCuaM6GI143xo62KO
unzip short_hair.zip
mv short_hair/ poses/
# for double ponytail girl
gdown https://drive.google.com/uc?id=1WNnGVuU0ZLyEn04HzRKzITXqib1wwM4Q
unzip double_ponytail.zip
mv double_ponytail/ poses/
We provide sample inputs of anime character sheets. You can also draw more by yourself. Character sheets need to be cut out from the background and in png format. Baidu Netdisk (password:RDxc)
# for short hair girl
gdown https://drive.google.com/uc?id=1r-3hUlENSWj81ve2IUPkRKNB81o9WrwT
unzip short_hair_images.zip
mv short_hair_images/ character_sheet/
# for double ponytail girl
gdown https://drive.google.com/uc?id=1XMrJf9Lk_dWgXyTJhbEK2LZIXL9G3MWc
unzip double_ponytail_images.zip
mv double_ponytail_images/ character_sheet/
We provide two ways: with web UI or via terminal.
- with web UI (powered by Streamlit)
streamlit run streamlit.py --server_port=8501
then open your browser and visit localhost:8501
, follow the instructions to genarate video.
- via terminal
mkdir {dir_to_save_result}
python -m torch.distributed.launch \
--nproc_per_node=1 train.py --mode=test \
--world_size=1 --dataloaders=2 \
--test_input_poses_images={dir_to_poses} \
--test_input_person_images={dir_to_character_sheet} \
--test_output_dir={dir_to_save_result} \
--test_checkpoint_dir={dir_to_weights}
ffmpeg -r 30 -y -i {dir_to_save_result}/%d.png -r 30 -c:v libx264 output.mp4 -r 30
@article{lin2022conr,
title={Collaborative Neural Rendering using Anime Character Sheets},
author={Lin, Zuzeng and Huang, Ailin and Huang, Zhewei and Hu, Chen and Zhou, Shuchang},
journal={arXiv preprint arXiv:2207.05378},
year={2022}
}