Error running evaluation on dual GPUs about minigpt-4

[matsuren]

I successfully run on dual RTX3090s.
Try with CUDA_VISIBLE_DEVICES=0 python demo.py --cfg-path eval_configs/minigpt4_eval.yaml.

[valine]

I successfully run on dual RTX3090s.
Try with CUDA_VISIBLE_DEVICES=0 python demo.py --cfg-path eval_configs/minigpt4_eval.yaml.

Curious, what's your VRAM usage between the cards? With CUDA_VISIBLE_DEVICES=0 I can run the model in 8bit model but my second card is idle. I'm hoping to run the full 16bit model split between my two cards.

[matsuren]

One is 16GB and the other is no usage. CUDA_VISIBLE_DEVICES=0 just limit visibility from python program. e.g., CUDA_VISIBLE_DEVICES=0: only GPU_id = 0 is visible, CUDA_VISIBLE_DEVICES=1: only GPU_id = 1 is visible, and CUDA_VISIBLE_DEVICES=0,1: both GPU_id=0 and 1 are visible.

If you want to run the inference using two cards then you need to do something like this