Comments (17)
Set the parameter device_map='auto' when load the LlamaForCausalLM.from_pretrained()
Replace the line in demo.py as: chat = Chat(model, vis_processor, device='cuda')
It can run on two RTX 2080Ti in my computer.It seems the model is implemented in two devices. But when doing the inference, the tensor flowed in two deivces and it will throw the two devices error. RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
(1) Load the LLaMA with device map to 'auto':
MiniGPT-4/minigpt4/models/mini_gpt4.py
Line 94 in 22d8888
device_map = 'auto'
(2) Modify the line below from 'cuda:{}'.format(args.gpu_id)' to 'cuda', It will automatically assign to device0 or device1 if you have two devices:
Line 64 in 22d8888
chat = Chat(model, vis_processor, device='cuda' )
(3) The "to device" can be removed from the line below because llama has been loaded to GPUs automatically:
Line 60 in 22d8888
model = model_cls.from_config(model_config)
(4) When encode the image, we may encode the image with CPU and assign the image embedding to GPU
image_emb, _ = self.model.encode_img(image.to('cpu'))
img_list.append(image_emb.to('cuda'))
The model should now work if you have multiple GPUs with low memory space.
from minigpt-4.
Yes i just restarted my cuda.
from minigpt-4.
Nope exactly same
from minigpt-4.
I think you van go ahead
from minigpt-4.
I also would like to know how to do this?
I have 2x3060 12gb so I could load the 13b model but it doesn't seem to be implemented
from minigpt-4.
I have same request.
from minigpt-4.
I have same request too.
from minigpt-4.
-
Set the parameter device_map='auto' when load the LlamaForCausalLM.from_pretrained()
-
Replace the line in demo.py as: chat = Chat(model, vis_processor, device='cuda')
It can run on two RTX 2080Ti in my computer.
from minigpt-4.
Set the parameter device_map='auto' when load the LlamaForCausalLM.from_pretrained()
Replace the line in demo.py as: chat = Chat(model, vis_processor, device='cuda')
It can run on two RTX 2080Ti in my computer.
It seems the model is implemented in two devices. But when doing the inference, the tensor flowed in two deivces and it will throw the two devices error.
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
from minigpt-4.
Traceback (most recent call last):
File "/home2/jainit/MiniGPT-4/demo.py", line 61, in
model = model_cls.from_config(model_config)
File "/home2/jainit/MiniGPT-4/minigpt4/models/mini_gpt4.py", line 243, in from_config
model = cls(
File "/home2/jainit/MiniGPT-4/minigpt4/models/mini_gpt4.py", line 90, in init
self.llama_model = LlamaForCausalLM.from_pretrained(
File "/home2/jainit/torchy/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2722, in from_pretrained
max_memory = get_balanced_memory(
File "/home2/jainit/torchy/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 731, in get_balanced_memory
max_memory = get_max_memory(max_memory)
File "/home2/jainit/torchy/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 624, in get_max_memory
_ = torch.tensor([0], device=i)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
I did all of these steps but i still get
from minigpt-4.
@JainitBITW Is it working now for you?
from minigpt-4.
@JainitBITW Did you do anything apart from @thcheung 's instruction?
Thanks anyway!
from minigpt-4.
What error you are getting
from minigpt-4.
I'm trying to run the 13 B model on multiple GPUs. The author has written they currently don't support multi-GPU inference. So , I want to be sure that it's possible to do inference on multiple GPUs before provisioning the ec2 instance.
from minigpt-4.
@JainitBITW @thcheung thanks it worked for me (8 bit). Have any idea how to do it for 16 bit (low resource = False) ?
It is throwing this error:
RuntimeError: Input type (float) and bias type (c10::Half) should be the same
from minigpt-4.
RuntimeError: Input type (float) and bias type (c10::Half) should be the same
I got through this error by setting vit_precision: "fp32"
in minigpt_v2.yaml
, but I didn't figure out what would need to be done to get the new input to also be fp16 (half precision) instead of making everything fp32.
from minigpt-4.
My solution is:
CUDA_VISIBLE_DEVICES=1 python demo_v2.py --cfg-path eval_configs/minigptv2_eval.yaml --gpu-id 0
from minigpt-4.
Related Issues (20)
- MiniGPT-v2 online demo is broken
- minigpt-v2 evaluation
- refcoco_eval.json HOT 2
- Colab notebook is no longer available HOT 2
- Vicuna 13b model, could not parse ModelProto from /home/ken/git-repos/vicuna/config.json HOT 3
- MiniGPTV2 Text to Text
- what does the checkpoint2(stage2) and checkpoint(stage3) mean in the Minigpt_v2?
- lego
- [Detection] Is there any way to output detection confidence?
- /media/dell/data1/ljw/code/test3/New/MiniGPT-4/eval_scripts/eval_vqa.py: 行 24: 未预期的符号 `(' 附近有语法错误
- The order of connection between word vectors and image vectors in prompt
- The Hugging Space app is not available.
- I want to use minigpt-v2 to achieve image search. What are the implementation ideas
- What does the parameter "data" printed during training mean HOT 1
- Where is the region_descriptions.json?
- loss becomes nan during training
- valid_step in training epochs
- Could not create share link. Please check your internet connection or our status page: https://status.gradio.app
- MiniGPT-4 Colab notebook
- demo.py error when starting chat
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from minigpt-4.