hanhung / tgnn

TGNN for Referring 3D Instance Segmentation

This is the code release for the paper Text-Guided Graph Neural Networks for Referring 3D Instance Segmentation.

Table of Contents 0. Package Versions 1. Dataset Download 2. Data Organization 3. Data Preprocessing 4. Pretrained Models 5. Training 6. Validation 7. To-Do 8. Acknowledgements

0. Package Versions

• Packages

  conda install -c conda-forge tqdm
  conda install -c anaconda scipy
  conda install -c conda-forge scikit-learn
  conda install -c open3d-admin open3d
  conda install pytorch==1.4.0 torchvision==0.5.0 -c pytorch
  conda install -c huggingface transformers
  

• Follow instructions from https://github.com/facebookresearch/SparseConvNet to download SparseConvNet.

1. Dataset Download

Scannet Download

For the Scannet Dataset please go to https://github.com/ScanNet/ScanNet and fill out the agreement form to download the dataset.

ScanRefer Download

For the ScanRefer Dataset please go to https://github.com/daveredrum/ScanRefer and fill out the agreement form to download the dataset.

Glove Embeddings

Download the preprocessed glove embeddings from ScanRefer.

2. Data Organization

scannet_data
|--scans

This Repository
|--glove.p
|--ScanRefer
    |--Files from ScanRefer download

3. Data Preprocessing

First store the point cloud data for each scene into pth files.

python prepare_data.py

Split the files into train and val folders.

python split_train_val.py

4. Pretrained Models

Please download the pretrained instance segmentation model and place into the folder like this.

This Repository
|--GRU
    |--checkpoints
        |--model_insseg-000000512.pth
|--BERT
    |--checkpoints
        |--model_insseg-000000512.pth

Pretrained model for referring model with gru encoder and place into the folder like this.

This Repository
|--GRU
    |--checkpoints
        |--gru
            |--models
                |--gru-000000032.pth

Pretrained model for referring model with bert encoder and place into the folder like this.

This Repository
|--BERT
    |--checkpoints
        |--bert
            |--models
                |--bert-000000064.pth

5. Training

Train the referring model with GRU encoder. (Note that we train with 2 GTX 1080Tis and Batchsize 8)

cd GRU/
python unet_gru.py

Train the referring model with BERT encoder. (Note that we train with 1 V100 and Batchsize 16)

cd BERT/
python unet_bert.py

6. Validation

Validate referring model with GRU encoder.

cd GRU/
python unet_gru_val.py

Validate referring model with BERT encoder.

cd BERT/
python unet_bert_val.py

7. Acknowledgements

Our dataloader and training implementations are modified from https://github.com/facebookresearch/SparseConvNet and https://github.com/daveredrum/ScanRefer, please go check out their repositories for sparseconvolution and 3D referring object localization implementations respectively. We would also like to thank the teams behind Scannet and ScanRefer for providing their dataset.

tgnn's Issues

Where could I get the file named 'glove.p'

When I try to run this work( 'python unet_gru.py' ),
I got 'FileNotFoundError: [Errno 2] No such file or directory: 'glove.p''

so, anyone could tell me where I could get this file.

No such file or directory: 'ScanRefer/ScanRefer_InstanceNames.json'

This file seems not in the offical ScanRefer dataset.

Some questions of TGNN.

Can I ask the number of parameters for the TGNN model, the memory required for training, and flops? Thanks！