sumo43 / loopvlm

run paligemma in real time

Hello 👋

• I am an undergraduate student in US
• I'm learning ML Engineering and DevOps

Contact

• email: [email protected]

loopvlm's Issues

How to detect multiple object

Hi, this is great using paligemma in real-time, but how to manage the model to detect multiple objects?

I give the prompt to 'detect blue car or black car', the model manages to detect blue or black car (here is the result),
but cannot detect multiple objects at a time.

output_video.mp4

Model loaded twice?

Why is the model loaded twice? I'm curious.

Once in 8 bit:

loopvlm/generate.py

Line 302 in e15de9b

model = _load_model(checkpoint_path, device, precision, use_tp)

Once in bfloat16:

loopvlm/generate.py

Line 321 in e15de9b

_model = PaliGemmaForConditionalGeneration.from_pretrained(