jiankai-sun / plate-pytorch

PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks

[ 📺 Website | 🏗 Github Repo | 🎓 Paper ]

Requirements

• PyTorch >= 1.4
• CrossTask Dataset

Data Preparation

Follow this folder structure to prepare the dataset:

.
└── crosstask
    ├── crosstask_features
    └── crosstask_release
        ├── tasks_primary.txt
        ├── videos.csv
        └── videos_val.csv

The data root is set here train_gpt.py.

How to Run

conda create -f environment.yml
bash srun.sh

Acknowledgement

We appreciate the following github repos a lot for their valuable code base implementations: joaanna/something_else, karpathy/minGPT.

Citation

@ARTICLE{PlaTe_RAL_2022,  
author={Sun, Jiankai and Huang, De-An and Lu, Bo and Liu, Yun-Hui and Zhou, Bolei and Garg, Animesh},  
journal={IEEE Robotics and Automation Letters},   
title={PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks},  
year={2022},  volume={7},  number={2},  pages={4924-4930},  
doi={10.1109/LRA.2022.3150855}}

plate-pytorch's Issues

About adding label with index 0 for some examples

In line 437 below, you seem to append label with index 0 for videos that have number of steps less than max (3 or 4). As far as I can tell, the index 0 is not reserved for a special action. Moreover, you don't have 0 in the beginning for the examples that have number of steps greater than max trajectory (3 or 4). May I know the reason for adding the label with index 0 for some but not other examples?
Thanks!

plate-pytorch/data_utils/data_loader_frames.py

Lines 424 to 437 in c388827

	if len(labels_matrix) > self.args.max_traj_len:
	idx = np.random.randint(
	0, len(labels_matrix) - self.args.max_traj_len)
	else:
	idx = 0
	frames = []
	for i in range(self.args.max_traj_len):
	frames.extend(
	images[min(idx + i, len(images) - 1)]) # goal
	frames = torch.tensor(frames)
	labels = []
	if idx - 1 < 0:
	labels.append([0])

The training process gets killed every time after several epochs.

What is the `bbox` folder supposed to contain for crosstask dataset?

Hi! Thanks for providing the code for your paper. I was trying to run the code after downloading the crosstask dataset. However, I did not find any folder called bbox_v0 or bbox in the dataset. May I know how I could get this folder? I tried looking at the comments in the code as well as the paper itself but I could not find any mention of such a folder. Any help in this regard would be greatly appreciated.

Question on the visual representation

Hi, I carefully walk through the code and find that the visual feature is pooled into 1 dimension, so the prediction is only based on the previous language (action) instead of the visual state. I just want to check if this is intended. Thanks