alexpashevich / e.t. Goto Github PK

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Home Page: https://sites.google.com/view/episodictransformer

License: MIT License

Python 42.59% C 48.47% Lex 0.45% Makefile 0.12% Shell 0.11% Yacc 3.26% PDDL 4.99%

e.t.'s People

Contributors

Stargazers

Watchers

Forkers

arjunakula trendingtechnology yizhouzhao yitongtseo siviltaram pushkalkatara samuel-fipps jesbu1 xyzhu123 georgiynefedov aaravmonga asmitahajra aishwaryaagrawal21 do-dong-park guanxinglu renos irelia-7

e.t.'s Issues

Stuck while rendering trajectory

Hi,
I'm now suffering from the code stucking and would ask some help to deal with it.

The problem command is as below:
python -m alfred.gen.render_trajs

From debugging, I found that the lines below makes it get stuck

E.T./alfred/gen/render_trajs.py

Lines 272 to 288 in 92ee237

    
           lock = threading.Lock() 
        
           if args.num_threads > 0: 
        
               # start threads 
        
               threads = [] 
        
               for worker_id in range(min(args.num_threads, len(traj_list))): 
        
                   thread = threading.Thread( 
        
                       target=start_worker, 
        
                       args=(worker_id, traj_list, args, lock, processed_files_path)) 
        
                   threads.append(thread) 
        
                   thread.start() 
        
                   time.sleep(1) 
        
               for thread in threads: 
        
                   thread.join() 
        
           else: 
        
               # run in the main thread 
        
               start_worker(0, traj_list, args, lock, processed_files_path) 
        
               return

More precisely, the code stuck while performing the lines below.

E.T./alfred/env/thor_env.py

Lines 65 to 76 in 92ee237

    
           super().reset(scene_name) 
        
           event = super().step(dict( 
        
               action='Initialize', 
        
               gridSize=grid_size, 
        
               cameraY=camera_y, 
        
               renderImage=render_image, 
        
               renderDepthImage=render_depth_image, 
        
               renderClassImage=render_class_image, 
        
               renderObjectImage=render_object_image, 
        
               visibility_distance=visibility_distance, 
        
               makeAgentsVisible=False, 
        
           ))

It stuck due to thorEnv so I searched some issue seems related.
askforalfred/alfred#120 (comment)

There is no error but the simulator just stop on some timestamp and don't move more than a day..
It would be very appreciate if you help

Share 45k synthetic trajectories

Thank you @alexpashevich for your great work.
When I followed your guide to Generate 45K trajectories with alfred.gen.generate_trajs, it mostly failed to generate trajectories in order to reproduce the result.
Rather than sharing the entire raw images, would you just share the traj_data.json with us? I think the total size of such files for 45k trajectories is around 3 GB. From these files, we can render the images easily.

Thank you so much!

Share pretrained checkpoint trainning process

Thank you @alexpashevich for your great work.

I am curious about the pretrained checkpoint training.

Can you share pretrained checkpoint trainning process?

Evaluating only sub-goals

I am trying to evaluate the pre-trained ET checkpoint on only the individual sub-goals, but am currently running into issues. Here is the command I ran:

$ python -m alfred.eval.eval_agent with eval.exp=pretrained \
          eval.checkpoint=et_human_pretrained.pth \
          eval.object_predictor=$ET_LOGS/pretrained/maskrcnn_model.pth \
          exp.num_workers=5 \
          eval.eval_range=None \
          exp.data.valid=lmdb_human\
          eval.subgoals=all

The only change I added from the command in the README is the eval.subgoals=all argument.
However, this eventually results in this error:

   File "/home/caisplusplus/tejas/ET/alfred/utils/eval_util.py", line 331, in load_language
    feat_numpy = dataset.load_features(*feat_args)
TypeError: load_features() takes 2 positional arguments but 4 were given

The error appears to be due to this line of code: when subgoal_idx is not None, it sends four arguments (including self) to this, which can only take two arguments. The AlfredDataset.load_features method does not seem to be defined to handle another subgoal_idx-related argument.

Is there some error here, or should I be running this script differently?

Not able to render on Colab

The below command:

python -m alfred.gen.render_trajs

works when i run it on my personal laptop, but does not seem to work on colab.

Are there any changes I can do so that I can get it working on colab.

Unable to download the pretrained checkpoints

I was not able to download the pretrained checkpoints using the following command:

wget http://pascal.inrialpes.fr/data2/apashevi/et_checkpoints.zip

visualization code

Thanks for your wonderful work! Would you release visualization codes for case visualization for its generated trajectory? like generating an MP4 or GIF file?

Questions about the pretrained model.

Thank you @alexpashevich for sharing this great work. I am here for asking whether you would release your different versions of pretrained models you published in your paper and on Alfred leaderboard.

How to generate the results on the testing set for the ALFRED leaderboard?

Thank you @alexpashevich for sharing this great work.

I was able to run on the validation set following the scripts in the README, but I didn't see any scripts that can generate the trajectories json files for the ALFRED leaderboard on the test set.

Thank you in advance for any assistance or guidance you can provide.

Trajectories were skipped

Thank you @alexpashevich for your great work
When I followed your guide to create an LMDB dataset with natural language annotations. It always report the error "string indices must b integers" in line 145 of the thor_env.py. The action "LookUp" seems not be an integer. This error would cause 7080 trajectories ere skipped. Would you like to tell us how to fixed it?

Fine-tuning?

Hi,

I am wondering if there's already a way (pipeline) to fine-tune this model to output a different set of actions using a different dataset? I figured I would ask before I edit the code myself.

Thank you!

Unable to download checkpoints with the given link

Can you please assist on how to resolve this issue when running the following command?

wget http://pascal.inrialpes.fr/data2/apashevi/et_checkpoints.zip

Resolving pascal.inrialpes.fr (pascal.inrialpes.fr)... 194.199.16.17
Connecting to pascal.inrialpes.fr (pascal.inrialpes.fr)|194.199.16.17|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-08-10 21:58:28 ERROR 403: Forbidden.

Question about the number of synthetic language demonstrations

Thanks for your great work!

I wanna ask why the number of synthetic language demonstrations is not equal to the number of expert trajectories in train split?
Since each trajectory corresponds to a single PDDL, and your synthetic language demonstration is just generated from PDDL, I supposed that the number of synthetic language demonstrations would be just the same as the number of expert trajectories in train split. However, the number of synthetic language instructions is about five times larger than the expert trajectories.
Is there anything I misunderstood or are you using other methods of data augmentation?
Thanks for your help!

render_trajs Seems to get stuck

When I run render_trajs, ai2thor loads just fine and everything but it gets stuck in the bathroom not doing anything.

Error in trying evaluation task

Thank you for your great work! @alexpashevich

Your paper was very interesting for me and i'm now trying to run your code in my local. But I have an issue when I try to run alfred.eval.eval_agent.

When I try to run the pretrained models evaluation code
python3 -m alfred.eval.eval_agent with eval.exp=pretrained eval.checkpoint=et_human_pretrained.pth eval.object_predictor=$ET_LOGS/pretrained/maskrcnn_model.pth exp.num_workers=5 eval.eval_range=None exp.data.valid=lmdb_human
FileNotFoundError came out and I cannot find which one is wrong.
The codes before this line were correctly run and worked well.

The picture below is the error I am currently suffer from.

And the pretrained directory looks as follow.

Should I copy the directory from somewhere or did I miss something before evaluate task?

Sorry for the poor question..

lack of file causes difficulty to reproduce the result

$ cd $ET_DATA
$ sh download_data.sh json_feat

i cannot find the download_data.sh

Stuck while rendering trajectory and Error in trying evaluation task

These issues are present in the current repo.

How to fix them:
Issue 1: Stuck while rendering trajectory:
#8
One of the modules in requirements.txt need to be a older version. I don't remember which.

Issue 2: Error in trying evaluation task:
#6
Happens when using cuda
You can either comment it out in model.util, I think you can add .cpu() to the cuda tensor like so: feat_extracted = feat_extracted.cpu()

	lock = threading.Lock()
	if args.num_threads > 0:
	# start threads
	threads = []
	for worker_id in range(min(args.num_threads, len(traj_list))):
	thread = threading.Thread(
	target=start_worker,
	args=(worker_id, traj_list, args, lock, processed_files_path))
	threads.append(thread)
	thread.start()
	time.sleep(1)
	for thread in threads:
	thread.join()
	else:
	# run in the main thread
	start_worker(0, traj_list, args, lock, processed_files_path)
	return

	super().reset(scene_name)
	event = super().step(dict(
	action='Initialize',
	gridSize=grid_size,
	cameraY=camera_y,
	renderImage=render_image,
	renderDepthImage=render_depth_image,
	renderClassImage=render_class_image,
	renderObjectImage=render_object_image,
	visibility_distance=visibility_distance,
	makeAgentsVisible=False,
	))