Comments (5)
Thanks for your interest! We are using 80G. I'm not sure if you mean the training stage or the inference. In case you are talking about training, I think a simple way to avoid OOM is to use a smaller batch size. You can set it in the training config files under the folder train_config/. I think maybe 8bit vicuna can also reduce memory usage. But we haven't tested this in the training stage yet. In case you mean inference, the current inference is run on a single card and it doesn't support model parallel in multi GPU yet. So multiple GPU will not help the OOM issue. Some methods discussed in this issue can reduce memory usage for inference dramatically. We are currently working on an official solution to make it run in a 24G memory GPU. Will return to you once it is finished
from minigpt-4.
discussed
How much GPU memory do I need if I use it for inference?
from minigpt-4.
Thanks for your interest! We are using 80G. I'm not sure if you mean the training stage or the inference. In case you are talking about training, I think a simple way to avoid OOM is to use a smaller batch size. You can set it in the training config files under the folder train_config/. I think maybe 8bit vicuna can also reduce memory usage. But we haven't tested this in the training stage yet. In case you mean inference, the current inference is run on a single card and it doesn't support model parallel in multi GPU yet. So multiple GPU will not help the OOM issue. Some methods discussed in this issue can reduce memory usage for inference dramatically. We are currently working on an official solution to make it run in a 24G memory GPU. Will return to you once it is finished
Thanks for your reply. I use 8xA100(40G) for training Vicuna with a GPU OOM problem now.
from minigpt-4.
@XXXKAY We update the default hyperparameter for inference and load Vicuna as 8bit by default now when you launch the demo. Under this setting, the memory cost is about 23GB. You can check the updated readme for more information
from minigpt-4.
@Unrealluver In your case I think you can set the batchsize per GPU smaller in minigpt4_stage1_pretrain.yaml. The default is 64, which cost about 70+ GB per GPU
from minigpt-4.
Related Issues (20)
- Confusion around normalising the bounding boxes between (0, 100) for fine-tuning.
- I couldn't find the checkpoint for MiniGPT-4 (llama2) in stage1
- 功能需求:提供图片输入API接口,向API接口传输图片的base64编码,以及需要GPT执行的工作,返回结果 HOT 1
- _pickle.UnpicklingError: invalid load key, '<'. HOT 1
- How to Finetune MiniGPT-v2 on 40G A100 GPUs? HOT 7
- throws the StopIteration error HOT 1
- Why lora is not used in minigpt4 version-1?
- the checkpoints of MiniGPT-v2 (after stage-3) and MiniGPT-v2 (online developing demo) have any differences? HOT 1
- Could not create share link. Please check your internet connection or our status page: https://status.gradio.app. HOT 1
- Has anyone ever tried to finetune miniGPT-4 on VQA tasks such as OKVQA?
- How to preprocess IconQA for evaluation? HOT 2
- urllib.error.HTTPError: HTTP Error 403: Forbidden
- Is is okay to use CrossEntropyLoss on referring or grounding?
- Hyperparameters for the VQA zero-shot inference
- Question: replacing the Llama 2 7b chat component with an equivalent Code Llama 7b
- cc_sbu_align datasets
- A load persistent id instruction was encountered, but no persistent_load function was specified. HOT 1
- minigpt4-demo
- demo.py 出错 HOT 3
- pip install accelerate仍然不可以 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from minigpt-4.