GPU Memory of A100 about minigpt-4

[TsuTikgiau]

Thanks for your interest! We are using 80G. I'm not sure if you mean the training stage or the inference. In case you are talking about training, I think a simple way to avoid OOM is to use a smaller batch size. You can set it in the training config files under the folder train_config/. I think maybe 8bit vicuna can also reduce memory usage. But we haven't tested this in the training stage yet. In case you mean inference, the current inference is run on a single card and it doesn't support model parallel in multi GPU yet. So multiple GPU will not help the OOM issue. Some methods discussed in this issue can reduce memory usage for inference dramatically. We are currently working on an official solution to make it run in a 24G memory GPU. Will return to you once it is finished

[XXXKAY]

discussed

How much GPU memory do I need if I use it for inference?

[Unrealluver]

Thanks for your interest! We are using 80G. I'm not sure if you mean the training stage or the inference. In case you are talking about training, I think a simple way to avoid OOM is to use a smaller batch size. You can set it in the training config files under the folder train_config/. I think maybe 8bit vicuna can also reduce memory usage. But we haven't tested this in the training stage yet. In case you mean inference, the current inference is run on a single card and it doesn't support model parallel in multi GPU yet. So multiple GPU will not help the OOM issue. Some methods discussed in this issue can reduce memory usage for inference dramatically. We are currently working on an official solution to make it run in a 24G memory GPU. Will return to you once it is finished

Thanks for your reply. I use 8xA100(40G) for training Vicuna with a GPU OOM problem now.

[TsuTikgiau]

@XXXKAY We update the default hyperparameter for inference and load Vicuna as 8bit by default now when you launch the demo. Under this setting, the memory cost is about 23GB. You can check the updated readme for more information

[TsuTikgiau]

@Unrealluver In your case I think you can set the batchsize per GPU smaller in minigpt4_stage1_pretrain.yaml. The default is 64, which cost about 70+ GB per GPU