deepseek-ai / deepseek-vl Goto Github PK
View Code? Open in Web Editor NEWDeepSeek-VL: Towards Real-World Vision-Language Understanding
Home Page: https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B
License: MIT License
DeepSeek-VL: Towards Real-World Vision-Language Understanding
Home Page: https://huggingface.co/spaces/deepseek-ai/DeepSeek-VL-7B
License: MIT License
If I want to run and use in ollama, I can import the GGUF version into ollama if it is available
Hi I want to finetune it on my custom OCR tasks, can you share finetuning script and also help in telling the best way to serve it very very fast.
Would love to see AMD/ROCm support. I know you guys are busy though. If I have the time I'll put in a pull request.
支持ollama部署吗
I installed this into a Python 3.8.0 Conda environment. with the "pip install - e ." -commands. Now, when I try to run it, I only get these lines:
(DeepSeekVL) C:\Users\Troc\DeepSeek-VL>python deepseek_vl/serve/app_deepseek.py
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
After that, the process just quits with zero explanation. The only sign that something happened is a slight bump in RAM usage. Did I do something incorrectly?
Traceback (most recent call last):
File "inference.py", line 56, in
outputs = vl_gpt.language_model.generate(
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/generation/utils.py", line 1527, in generate
result = self._greedy_search(
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
outputs = self(
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 1196, in forward
outputs = self.model(
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 990, in forward
causal_mask = self._update_causal_mask(attention_mask, inputs_embeds, cache_position)
File "/root/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 1076, in _update_causal_mask
causal_mask = torch.triu(causal_mask, diagonal=1)
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'
python=3.8
accelerate 0.28.0
aiofiles 23.2.1
altair 5.2.0
annotated-types 0.6.0
anyio 4.3.0
attrdict 2.0.1
attrs 23.2.0
Brotli 1.0.9
certifi 2024.2.2
charset-normalizer 2.0.4
click 8.1.7
colorama 0.4.5
contourpy 1.1.1
cycler 0.12.1
deepseek_vl 1.0.0 /app
einops 0.7.0
exceptiongroup 1.2.0
fastapi 0.110.0
ffmpy 0.3.2
filelock 3.13.1
fonttools 4.50.0
fsspec 2024.3.1
gmpy2 2.1.2
gradio 3.48.0
gradio_client 0.6.1
h11 0.14.0
httpcore 1.0.4
httpx 0.27.0
huggingface-hub 0.22.0
idna 3.4
importlib_metadata 7.1.0
importlib_resources 6.4.0
Jinja2 3.1.3
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
latex2mathml 3.77.0
Markdown 3.4.1
MarkupSafe 2.1.3
matplotlib 3.7.5
mdtex2html 1.3.0
mkl-fft 1.3.8
mkl-random 1.2.4
mkl-service 2.4.0
mpmath 1.3.0
networkx 3.1
numpy 1.24.3
orjson 3.9.15
packaging 24.0
pandas 2.0.3
pillow 10.2.0
pip 23.3.1
pkgutil_resolve_name 1.3.10
psutil 5.9.8
pydantic 2.6.4
pydantic_core 2.16.3
pydub 0.25.1
Pygments 2.12.0
pyparsing 3.1.2
pypinyin 0.50.0
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-multipart 0.0.9
pytz 2024.1
PyYAML 6.0.1
referencing 0.34.0
regex 2023.12.25
requests 2.31.0
rpds-py 0.18.0
safetensors 0.4.2
semantic-version 2.10.0
sentencepiece 0.1.96
setuptools 68.2.2
six 1.16.0
sniffio 1.3.1
starlette 0.36.3
sympy 1.12
tiktoken 0.5.2
timm 0.9.16
tokenizers 0.15.2
toolz 0.12.1
torch 2.0.1
torchaudio 2.0.2
torchvision 0.15.2
tqdm 4.64.0
transformers 4.39.1
triton 2.0.0
typing_extensions 4.9.0
tzdata 2024.1
urllib3 2.1.0
uvicorn 0.29.0
websockets 11.0.3
wheel 0.41.2
zipp 3.18.1
Is base model is stage1 or stage2 model? and chat is stage3 model?
User: <image_placeholder>描述这张图片
Assistant: 这张图片捕捉了自动驾驶汽车在城市街道上导航的瞬间。汽车是图片的中心焦点,位于画面**,正朝着画面的右侧移动。它正在行驶的街道上标有黄色线条,表明了它的路径。
在汽车的两侧,可以看到两个数字标记。左边的标记显示了汽车的速度,而右边的标记显示了汽车的距离。这些标记为汽车提供了方向和距离的清晰指示。
在汽车的右侧,有一个蓝色的图标。这个图标可能代表一个特定的功能或设置,如转向辅助或车道保持辅助。
在汽车的左侧,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个黄色的图标。这个图标可能代表一个特定的功能或设置,如自动变道辅助或车道保持辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道辅助。
在汽车的上方,有一个灰色的图标。这个图标可能代表一个特定的功能或设置,如自动泊车或自动变道
ms-swift大模型训练框架已经支持了deepseek-vl系列模型的推理和微调~
the SAM using different means compare to Siglip,
but from the code, the VLIChatProcessor are just using 1024 as size, imagenet_mean as input, this is right for SAM, but not right for siglip.
the processed tensors are directly sending to HybridTower without any further process.
Why doing this? Is the correct?
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'
torch 2.0.1
torchaudio 2.0.1
torchvision 0.15.2
请教一下,paper里提到的LLaVA1.6-GPT4V是什么数据集呢?
Hi,
I read your report, and I think the pipeline is very similar to Vary. I have a question:
Does the sam-branch use the Vary initialization for dense OCR? Based on my experiments, the vision latent output by the original Sam is noisy for text-latent-based LLM.
Sorry for this dummy question but I did search for some answers and try before.
Using llama.cpp
python ./convert-hf-to-gguf.py \
../../deepseek-vl-7b-chat \
--outtype f16 \
--outfile ../../deepseek-vl-7b-chat/deepseek-v1-7b-chat.gguf
returned
Loading model: deepseek-vl-7b-chat
Traceback (most recent call last):
File "/home/zhangyuanfeng/software/ollama/llm/llama.cpp/./convert-hf-to-gguf.py", line 2099, in
main()
File "/home/zhangyuanfeng/software/ollama/llm/llama.cpp/./convert-hf-to-gguf.py", line 2079, in main
model_class = Model.from_model_architecture(hparams["architectures"][0])
File "/home/zhangyuanfeng/software/ollama/llm/llama.cpp/./convert-hf-to-gguf.py", line 215, in from_model_architecture
raise NotImplementedError(f'Architecture {arch!r} not supported!') from None
NotImplementedError: Architecture 'MultiModalityCausalLM' not supported!
So is there any feasible method? Thx.
python deepseek_vl/serve/app_deepseek.py
Its not working on colab notebook. Need a fix !!
Congratulations to DeepSeek for the wonderful work. I wonder if there is a script for fine-tuning DeepSeek-VL? Thanks!
not support python 3.11.8?
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
'FieldInfo' object has no attribute 'required'
transformers==4.38.2
请问如何解决
Does DeepSeek-VL series support input of multiple images? This doesn't seem to be stated in the paper, but images
in the example script are list
, which seems to be supported.
Hey, thanks a lot for sharing this great accomplishment with the community!
I have just tried running the cli_chat
on Python3.11 and I get ImportError: cannot import name 'Mapping' from 'collections'
at https://github.com/deepseek-ai/DeepSeek-VL/blob/main/deepseek_vl/models/modeling_vlm.py#L21.
I believe this is due to attrdict being broken with python3.10+ https://stackoverflow.com/questions/72361026/how-can-i-get-attrdict-module-working-in-python.
Easiest thing would be to update Readme/requirements to specify >=python3.8, <3.10.
请问DeepSeek-VL最多支持多少张图片输入?
For instance:
thanks.
I hope to get the output.scores, but after generate, it have just returned a Torch.Tensor. How to rewrite the generate function ?
How did you unify the format of pretraining dataset? During supervised fine tuning stage, the training data are curated as question and answer pairs. For caption or detection dataset, I want to know if they follow the same format as sft data, and how to collect questions for these data as they originally only contains ground truth like caption or boxes?
Hi, How Can I replace this model's language model with another model and fine-tune it?
and can't i fine tune this model without shift?
Thanks the nice work~!
The image embedding has the size of torch.Size([1, 628, 4096]) when input one image.
How to get the one feature vector of size 4096 or some else?
Thanks in advance!
Hi! Thank you for releasing this multimodal model. First test are impressive. Even 1.3B is good for its size.
It is just that 7b version in full precision is still taxing on personal HW we have at home.
Would it be possible to quantize it to int4 like Qwen did with their Qwen-VL-Chat-Int4?
I think it would be best if you could do it and put it in your HF repo so community can use it.
If not, maybe you could give us some guidelines how to do it.
Hi! I'd love vLLM and exllamav2 support and wanted to help with that. Is the LLM side of the model pure llama2 architecture (not counting the lin. proj. layers)?
Thank you in advance
I tried to run this model in Gradio GUI on Windows 10 but I had a few issues.
I modified Inference.py (the one in deepseek_vl\serve) a bit to fix those issues.
I also made sure that my torch was installed with cuda 11.8 and not cpu-only mode.
So, if someone else runs into problems when running this model on 24GB of VRAM, this issue might help you.
Installation instructions (assuming you're already in a virtual env, which you should be using)
git clone https://github.com/deepseek-ai/DeepSeek-VL
cd DeepSeek-VL
pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -e .[gradio]
##replace inference.py file in the repo in folder deepseek_vl/serve with the one provided by me
##if you have the model downloaded locally, maybe change the path in app_deepseek.py to a local one
python deepseek_vl/serve/app_deepseek.py
Here's the Inference.py that works for me
https://gist.github.com/adamo1139/511f63c01c6088d7747f47628ffc970c
I will be closing it down, just want to leave a trace that will hopefully save some time for others.
DeepSeek-VL 7B seems to be the awesome vision model that nobody is talking about. It is better than Llava 1.6 34B in VQA and OCR. It consistently return JSON when asked and just seems to have a great textual capability. Any recommendation on how to wrap DeepSeek-VL in a fast inference server and take for spin at scale ?
There are some issues opened here:
InternLM/lmdeploy#1321
sgl-project/sglang#297
Is this code "optimal" for batched inference and preprocessing?
We've deployed the model,but An error is reported during the reasoning process. Can't multi-graphics inference,How to solve it
Hello, can you provide the training cost (e.g. training hours & training samples) of deepseek-vl-1.3B?
如题。
目前开源的多模态大模型似乎都只有有限的agent能力,要么不能完全理解文字,要么图像识别能力无法兼顾,要么没有任务规划能力,要么缺乏函数调度或者解释器能力。
deepseek目前的模型里,67B有文本agent的潜力,但是太大。
很期待deepseek在agent领域的发力。
Seems like parts of the code explicitly call cuda functions, so you can't just switch it to MPS for Macs. Any roadmap for supporting Macs?
Io.py
def load_pretrained_model(model_path: str):
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer
vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()
error
File "/Users/user/projects/ai/DeepSeek-VL/deepseek_vl/utils/io.py", line 37, in load_pretrained_model
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2528, in cuda
return super().cuda(*args, **kwargs)
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 911, in cuda
return self._apply(lambda t: t.cuda(device))
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 802, in _apply
module._apply(fn)
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 802, in _apply
module._apply(fn)
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 802, in _apply
module._apply(fn)
[Previous line repeated 2 more times]
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 825, in _apply
param_applied = fn(param)
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 911, in <lambda>
return self._apply(lambda t: t.cuda(device))
File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/cuda/__init__.py", line 293, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
LMDeploy, as an AI deployment platform supporting multiple backend services, has always been committed to providing fast and stable AI model deployment services.
Now, it supports accelerating the inference and service of DeepSeek VL models. Check InternLM/lmdeploy#1335
ValueError: The checkpoint you are trying to load has model type multi_modality
but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
如果用 python3.10 + CUDA11.7/CUDA11.8 + torch2.0.1,运行 inference.py 会报错如下:
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
改为 python3.8 后解决。
当我运行代码 vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
时会报错 TypeError: Object of type AlignerConfig is not JSON serializable
我的模型文件是huggingface上的1.3B-chat版本
when I run code vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
it show error like TypeError: Object of type AlignerConfig is not JSON serializable
my model is 1.3B-chat version download from huggingface
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.