opengvlab / instruct2act Goto Github PK

Hello and thank you for open-source your work. I would like to ask you about the "add cuda device option in function build_sam()" you mentioned, which parts of the SAM source code have been modified?

I followed the link as described on the README.md but I was unable to find the CLIP checkpoints on the link provided, also the segment_anything package is not included it the environment.yaml

Hi，I follow your "readme" and try to recurrent your program. But when I run the "robotic_anything_gpt_online_eval.py", it warn me that "No module named 'easydict'". So I write the issue and try to find your help.
Thanks!

Hi,

In the 90 line of engine_robotic.py, "tokenizer = open_clip.get_tokenizer("/data/openclip_tokenizer", direct_load=True)", where can I find the "openclip_tokenizer"?
The code assumes the openclip_tokenizer exists in the /data directory, but this file does not exist in the OpenCLIP repository. In laion/CLIP-ViT-H-14-laion2B-s32B-b79K, is it tokenizer.json?

In OpenCLIP, I see that openclip_tokenizer may be a function or class rather than a file. If so, the naming of this line of code could be improved, e.g. "tokenizer = open_clip.get_tokenizer(model_path="/data/...", direct_load=True)".

Thanks!

Hi!

There is another question. When I run the robotic_anything_gpt_online_eval.py ,it fails with Pretrained weights not found for model ViT-H-14. How can I get this file?

Thanks!

Hi,

I successfully run the visual window of Vimabench. But I dont know how to make it move after I read the "readme". Could you please give me any suggestions.

Thanks

Hi，

I run the offline mode these days. Can I get the success rate form of offline mode to compare with my result.

Thanks!

(vima) csc@csclab:~/LLM$ /home/csc/anaconda3/envs/vima/bin/python /home/csc/LLM/Instruct2Act/robotic_anything_gpt_online_eval.py
pybullet build time: May 20 2022 19:45:31
[INFO] 17 tasks loaded
/home/csc/anaconda3/envs/vima/lib/python3.9/site-packages/scipy/init.py:155: UserWarning: A NumPy version >=1.18.5 and <1.25.0 is required for this version of SciPy (detected version 1.26.4
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
Loading SAM...
Traceback (most recent call last):
File "/home/csc/LLM/Instruct2Act/robotic_anything_gpt_online_eval.py", line 18, in
from engine_robotic import *
File "/home/csc/LLM/Instruct2Act/engine_robotic.py", line 47, in
build_sam_func[sam_idx](
TypeError: build_sam_vit_h() got an unexpected keyword argument 'device'

do not know how to solve it.

Hi, thanks for your excellent work!
I wonder whether there is a rotation in the Rearrange task?
If yes, how do you calculate the degree? Thanks!

Hello,

Thank you for your awesome work!
I meet a problem that the dragged_obj_1 can not be put into the base_obj correctly. I have checked the output of CLIPRetrieval() function. In my testing, the coordinate of dragged_obj_1 is 2, and base_obj is 3, and sometimes they are both 2. Could you give me some suggestions what causes the problem. May be I should try some larger model? I use sam_vit_b_01ec64 the save the memory. But I do not think it is the problem of model, since the robot can pick the dragged_obj_1 every time accrurately, just put it down to the same position rather than the position of base_obj.

Looking forward to you reply!

Hi,
thanks for your excellent work.

I ckpts for CLIP and SAM prepared.
robotic_anything_gpt_online_eval.py
is executed,

On the terminal
openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.

Could you please tell me what the solution is?

Hi,
After launching the GUI interface, I expected the robotic arm to move or provide some form of feedback. However, I am not observing any movement or receiving any prompts or error messages.
Thank you for your time and support.

Hi,
thanks for your excellent work.

1. I tried to run the _robotic_anything_gpt_online.py. However, even with a proxy, I still got the error "_openai.error.AuthenticationError: ". The network conditions cannot be improved at the moment.
2. I tried to use the LLAMA-adapter to implement Instruct2Act. I would like to know how to use the LLAMA-adapter and robotic_anything_offline.py to implement Instruc2Act. Will this part of the code be open source later?
   Any help would be much appreciated.
   Bests

Hi, thanks for your work

What if I just want to get the result after using SAM and Openclip, without performing any robot actions

Which files should i modify and run

Thanks!

hi, thanks for your excellent work!

when i have installed all packages and run robotic_anything_gpt_online_eval.py, error happens:

I am a beginner in LLM and robotics, so i really don't know how to fix it , can you help me?

The OpenGL renderer open for the first task, completed it successfully. However all the subsequent task fail with the error shown above.