How to obtain multiple hotspots? about vrb

[ubless607]

@Tom-Huang Sent!

[jun7-shi]

@ubless607 thanks so much.
Using the pretrained weights you provided, i could reproduce the results of multiple hotspots.

[ubless607]

@ZY199301 sent!

[Kimagure7]

Hi @ubless607 or anyone in the above thread, could you send the weights to me? Thank you very much in advance! :) My email is: [email protected]

sent!

[Hotsburp]

@ubless607

Yup, i will try this! May i know what is the cuda and nvidia driver versions that you were using for your case?

Update: I have managed to run the demo.py script, but i used the commit version "https://github.com/IDEA-Research/GroundingDINO/tree/da9f1c0751935e9284890572069f4b2fef4cbfdc" instead else the "supervision" versions will conflict for pillow.

I have a CUDA out of memory error so i probably need to source for a better gpu before proceeding.

[Hotsburp]

@ubless607

Thanks for the info on the setup! I think it would be best to run GroundingDino using the https://github.com/IDEA-Research/GroundingDINO/tree/da9f1c0751935e9284890572069f4b2fef4cbfdc version (follow the GPU steps as well) as it is at the same time when VRB was pushed (~April 2023). LangSegAnything can be the latest version and modified based on your recommendations. Just need to make sure supervision==0.6.0, pillow==9.3.0, and CUDA==1.7 as the system will highlight dependency conflicts.

Anyway, I have managed to run the demo.py code on the images by the authors and also some generalized images. Looks like the generalization is not the best as it is limited to a few items (cup, drawer, potlid, microwave) on inference.py and performs pretty inconsistently on certain viewpoints. But i think the author has addressed that it does not always work well in the #3 (comment) issue.

Also, do you happen to have the actual input image that you were using above? I wanted to test and maybe see if i have the same results as you but i can't find it on the project page.

The project has a lot of potential and i hope @shikharbahl releases the training code soon for future works!

[Hotsburp]

Hey @ubless607, yeap i tried out a few more use cases but it is sensitive to the image size and dimensions. As mentioned, I got good results only for certain tasks and angles but there are also a lot of wrong trajectories too haha.

Would be good that they release more tasks and also how this can be used for 3d pose/state beyond 2d image coordinates.

What about yourself, did you explore more too?

[ubless607]

@Hotsburp I didn't try on this much further :(

[ubless607]

@codepk37 Sent!

[ubless607]

@ashiqfauna Sent!

[ubless607]

@yolo01826 Can you write your e-mail address?

[ubless607]

@yolo01826 sent!

[jun7-shi]

@ubless607 hi, could you please send me the pretrained weights as well? [email protected] , thanks!

[ubless607]

@jun7-shi Sent!

[ZY199301]

hi, could you please send me the pretrained weights as well? [email protected], thank you very much!

[JaneLin-0409]

@ubless607 hi, could you please send me the pretrained weights as well? [email protected] , thanks!

[wadeKeith]

@ubless607 Could you please send me the pretrained weights as well? [email protected], thank you very much!

[ubless607]

@JaneLin-0409 sent!

[ubless607]

@wadeKeith sent!

[fuyh20]

@ubless607 I also need the pretrained weights. Could you send me it? [email protected], thanks!

[Byo0923]

@ubless607
I also need the pretrained weights too.
Could you please send me the pretrained weights as well? , [email protected] , thanks!

[ubless607]

@fuyh20 Sent!

[ubless607]

@Byo0923 Sent!

[Byo0923]

@ubless607
I was able to download the checkpoints. Thank you very, very much. I will try it.

[Kimagure7]

I also need the pretrained weights too. Could you please send me the pretrained weights as well? [email protected] , thanks! @ubless607

[ubless607]

@Kimagure7 Sent!

[fuyh20]

@ubless607 Thanks for the model weights. And I have already used it. You can set the manual seed in order to get different outputs, and the output is random.

[YujpVc]

I also need the pretrained weights, could you send me well? [email protected], thanks

[nys601350111]

@ubless607 I also need the pretrained weights, could you send me well? [email protected]](mailto:[email protected]) thanks!

[ubless607]

@YujpVc sent!

[ubless607]

@nys601350111 sent!

[YujpVc]

thanks a lot!

…

---Original--- From: ***@***.***> Date: Tue, Jul 9, 2024 00:08 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [shikharbahl/vrb] How to obtain multiple hotspots? (Issue #6) @YujpVc sent! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Hotsburp]

Hi @ubless607 or anyone in the above thread, could you send the weights to me? Thank you very much in advance! :) My email is: [email protected]

[Hotsburp]

Did anyone face any issue pip installing the lang-segment-anything portion from the "pip install -U git+https://github.com/luca-medeiros/lang-segment-anything.git" command?

Mine seems to be stuck at this portion for really really long.

Collecting lightning<3.0.0,>=2.0.1
Using cached lightning-2.0.2-py3-none-any.whl (1.8 MB)
Using cached lightning-2.0.1.post0-py3-none-any.whl (1.8 MB)
Using cached lightning-2.0.1-py3-none-any.whl (1.8 MB)

Not sure if i am doing anything wrong or it just takes super long to install. Thanks in advance again!

I will open a new issue but just wanted to ask here since some of you may be active.

[ubless607]

@Hotsburp I don't have any problem.
Did you try on new conda envrionment? / Did you update pip to the lastest version?

[Hotsburp]

Hi @ubless607, thanks for getting back! Yup i tried on new conda environment and also updating pip.

I think the issue is with the GroundingDino that LangSegmentAnything uses because there are a lot of issues with the version conflict of packages recently due to GroundingDino updates.

May i know which commit version (date or hash) of the LangSegmentAnything did you use or did you just use the latest version at that point of time? This is because this affects the version of GroundingDino used and i would also like to track back the date of the version that VRB used.

While i managed to now install somewhat properly, i had to manually find an older version of both LangSegmentAnything and GroundingDino.

But by doing so, i had this error when running demo.py using the weights:
=> _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight'])

This results in:
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

I would appreciate if you all have any insights! :)

[ubless607]

@Hotsburp It has been a while I tested this repository. Can you check this issue?
luca-medeiros/lang-segment-anything#66

[Hotsburp]

Yeap, i saw that too. No worries, i will try it out again. Thanks for you help!

I will let you know if i find any insights to your original issue of obtaining multiple hotspots.

[ubless607]

@Hotsburp

1. Git clone this fork version and install GroundingDINO
   https://github.com/IDEA-Research/GroundingDINO/tree/df5b48a3efbaa64288d8d0ad09b748ac86f22671

2. Git clone lang-segment-anything

3. Edit pyproject.toml

• huggingface-hub = "^0.16.4, <0.17.0"
• transformers = "4.27.4"
• lightning = "2.0.1"
• erase GroundingDINO line

Can you try?

[64327069]

@ubless607 I also need the pretrained weights, could you send me well? [email protected], thanks

[ubless607]

@64327069 Sent!

[ubless607]

@Hotsburp I am glad that you were able to reproduce the environment setting.

[Hotsburp]

@ubless607 During running demo.py on this image. Do you face the error "ValueError: tile cannot extend outside image"? I think i may be something to do with image crop as the error comes from the input_img in Line 84 of inference.py.

If i input img instead of input_img in Line 84, i get this:

I guess there is still multiple hotspots but the script is sensitive to the cropped image and its dimensions.

[bethalageetachandraraju]

can anyone from the above thread send me pretrained weights please, my email: [email protected]
@ubless607 @Byo0923 @wadeKeith @JaneLin-0409 @64327069 @Kimagure7

[ojh6404]

can anyone from the above thread send me pretrained weights please, my email: [email protected] @ubless607 @Byo0923 @wadeKeith @JaneLin-0409 @64327069 @Kimagure7

Could anyone send me pretrained weights, too? my email : [email protected]

[achrxme-1838]

could anyone from the above thread send me pretrained weights too please, my email: [email protected]
I'll be very appreciate it!!
@ubless607 @Byo0923 @wadeKeith @JaneLin-0409 @64327069 @Kimagure7

[ubless607]

@achrxme-1838 @ojh6404 @bethalageetachandraraju Sent!

[achrxme-1838]

Thank you-!!!! 2024년 8월 19일 (월) 오후 8:44, ubless607 ***@***.***>님이 작성:

…

@achrxme-1838 <https://github.com/achrxme-1838> @ojh6404 <https://github.com/ojh6404> @bethalageetachandraraju <https://github.com/bethalageetachandraraju> Sent! — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A74KOHTIR6XHY43ULZCUAMLZSHLAJAVCNFSM6AAAAABFBO4522VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJWGM3TIMZXG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ubless607]

@ubless607 During running demo.py on this image. Do you face the error "ValueError: tile cannot extend outside image"? I think i may be something to do with image crop as the error comes from the input_img in Line 84 of inference.py.

If i input img instead of input_img in Line 84, i get this:

I guess there is still multiple hotspots but the script is sensitive to the cropped image and its dimensions.

I didn't encounter that error message. Have you done any additional work on this?

[Tom-Huang]

could anyone from the above thread send me pretrained weights too please, my email: [email protected]

[codepk37]

@ubless607 @Byo0923 @wadeKeith @JaneLin-0409 @64327069 @Kimagure7 Could anyone please send me the pretrained weights aswell? (mailto: [email protected])

[ashiqfauna]

@ubless607 Could you please send me the pretrained weights as well? (mailto: ashiq).

Thanks :)

[dtbpkmte]

@ubless607 Can I have the weights please? My email address is [email protected]. Thank you!

[ubless607]

@dtbpkmte Sent!

[ZTE-dzq]

@ubless607 I also need the pretrained weights, could you send me? My email address is [email protected] , thanks!

[prac-101]

@ubless607 @Byo0923 @wadeKeith @JaneLin-0409 @64327069 @Kimagure7 I also need the pretrained weights, could anyone please send me the pretrained weights too please? My email address is [email protected], thanks a lot!