Car → Police	Car → Sketch
Motorbike → Tiger	Shoe → Mossy

You can make any other interesting stylish 3D object!

• On GET3D Repository: You can just clone this repo into your own computer

git clone https://github.com/nv-tlabs/GET3D.git && cd GET3D
git clone https://github.com/studio-YAIVERSE/studio-YAIVERSE.git && cd studio-YAIVERSE

• Without GET3D Repository: Use GitHub submodule, so you can clone GET3D repository automatically.

git clone --recursive https://github.com/studio-YAIVERSE/studio-YAIVERSE.git
cd studio-YAIVERSE
# if you are updating an existing checkout
git submodule sync && git submodule update --init --recursive

• Docker container

cd docker
docker build -f Dockerfile -t studio-yaiverse:v1 .
cd ..

• Anaconda environment

conda create -n get3d python=3.8
conda activate get3d
pip install -r requirements.txt

For GET3D + NADA, you need pretrained model's checkpoint. You can set downloaded ckpt path at yaml file.

• Car, Chair, Table, Motorbike → link

• Fruits, Shoe → link

And finally the directory hierarchy is configured as,

GET3D
├── studio-YAIVERSE (= Your Working Directory)
|	├── assets
|	├── docker
|	├── experiments
|	|	├── *.yaml
|	|	└── ....
|	├── legacy
|	├── results
|	|	├── default
|	|	|	  ├── checkpoint
|	|	|	  ├── sample
|	|	|	  └── default_date.log
|	|	└── ....
|	├── scripts
|	|	├── README.md
|	|	├── clip_save.py
|	|	├── image_to_video.py
|	|	└── render_shapenet_multigpu.py
|	├── sample_img
|	├── LICENSE
|	├── README.md
|	├── clip_loss.py
|	├── dist_util.py
|	├── functional.py
|	├── model_engine.py
|	├── nada.py
|	├── requirements.txt
|	├── text_templates.py
|	└── train_nada.py
├── 3dgan_data_split
├── data
└── ....

or,

studio-YAIVERSE (= Your Working Directory)
├── GET3D (submodule)
|	├── 3dgan_data_split
|	├── data
|	└── ....
├── assets
├── docker
├── experiments
|	├── *.yaml
|	└── ....
├── legacy
├── results
|	├── default
|	|	  ├── checkpoint
|	|	  ├── sample
|	|	  └── default_date.log
|	└── ....
├── scripts
|	├── README.md
|	├── clip_save.py
|	├── image_to_video.py
|	└── render_shapenet_multigpu.py
├── sample_img
├── LICENSE
├── README.md
├── clip_loss.py
├── dist_util.py
├── functional.py
├── model_engine.py
├── nada.py
├── requirements.txt
├── text_templates.py
└── train_nada.py

If you want to train the code, please refer to the training script below.

$ # working directory: studio-YAIVERSE
$ python train_nada.py --config_path='experiments/{}.yaml' --name='{}' --suppress

optional arguments
	--config_path             select yaml file to run (in experiments folder)
	--name                    choose any name you want for log file name (optional)
	--suppress                store only latest & best pkl file

EX)
$ python train_nada.py --config_path='experiments/car_police_example.yaml' --name='car_police' --suppress

When you open yaml file, you could see many trainable parameters and configs.

Among them, below are some important parameters you could change as you conduct an experiment.

We provide some yaml files as examples.

Global Config

GET3D config

NADA config

• You may have to change GET3D model's checkpoint to apprpriate one, by clip_loss value of given prompt text or image.
• Function match_checkpoint, in legacy/backend_input_checkpoint_map.py, returns best combination of source and target text. You can parse best checkpoint from it, like given example code:

  ...
  get3d_generator: GeneratorDMTETMesh
  
  from legacy.backend_input_checkpoint_map import match_checkpoint, parse_checkpoint_path
  src, trg = match_checkpoint(
      input_text,
      input_image_path,
      embedding_group_pt_path,
      device
  )
  checkpoint_path = parse_checkpoint_path(src, trg, args.checkpoint_root))
  get3d_generator.load_state_dict(torch.load(checkpoint_path))
  ...
  # Your inference code with GET3D...

• embedding_group_pt_path is the path of "embedding_group_pt", which could be built from legacy/generate_embedding_group_pt.py script.

• You can see end-to-end inference code - from text or image to 3d binary files - in backend's studio_YAIVERSE.apps.main.pytorch package. See here.
• Since pytorch package is implemented with related import method, you can copy pytorch folder and just harness it, like given example:

  from pytorch import init, inference
  
  YOUR_CONFIG: dict  # Write your config refer to init.__doc__
  init(YOUR_CONFIG)
  
  result = inference("object_name", "your_text_prompt")
  
  with open("result.glb", "wb") as fp:
      fp.write(result.file.getvalue())  # result.file: BytesIO
  
  with open("thumbnail.png", "wb") as fp:
      fp.write(result.thumbnail.getvalue())  # result.thumbnail: BytesIO

• We provide our checkpoint for some text templates, in backend repository. See checkpoint release.

• We provide scripts/clip_save.py to avoid 'connection reset by peer' error from CLIP library, which accidentally stops the runtime.

1. Do python scripts/clip_save.py, and then you can get clip-cnn.pt / clip-vit-b-16.pt / clip-vit-b-32.pt
2. Change clip.load() argument as follows (Note that this is used at clip_loss.py)
   • clip.load('RN50') → clip.load('/PATH/TO/clip-cnn.pt')
   • clip.load('ViT-B/16') → clip.load('/PATH/TO/clip-vit-b-16.pt')
   • clip.load('ViT-B/32') → clip.load('/PATH/TO/clip-vit-b-32.pt')

• We provide scripts/image_to_video.py to convert image sequences into a video.
• Do python scripts/image_to_video.py -i /PATH/TO/INPUT_IMAGE_DIR -o /PATH/TO/OUTPUT_VIDEO_FILE
• This code is used for visualizing our result like Introduction section.

• Rendering ShapeNet is required for training GET3D model with new category.
• We provide scripts/render_shapenet_multigpu.py to render ShapeNet with multi-GPU.
• This code is same with GET3D/render_shapenet_data/render_all.py, but makes blender rendering faster.
• Copy this script's content to GET3D/render_shapenet_data/render_all.py and run it.

• Please note that this is not official code by GET3D authors. There may be differences in detail from the original.
• TAPS3D branch contains our incomplete trial of GET3D training code + TAPS3D training code.
• If you have any question for our team project, don't hesitate to leave an issue or email to [email protected]. Thanks.