Comments (2)
请问我用这个demo在linux运行v2需要安装哪些环境?谢谢 import torch from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
model_name = "deepseek-ai/DeepSeek-V2-Chat" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
max_memory
should be set based on your devicesmax_memory = {i: "75GB" for i in range(8)}
device_map
cannot be set toauto
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, device_map="sequential", torch_dtype=torch.bfloat16, max_memory=max_memory, attn_implementation="eager") model.generation_config = GenerationConfig.from_pretrained(model_name) model.generation_config.pad_token_id = model.generation_config.eos_token_id
messages = [ {"role": "user", "content": "Write a piece of quicksort code in C++"} ] input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") outputs = model.generate(input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True) print(result)
同问
from deepseek-v2.
需要安装的python库包括:torch
,transformers
,accelerate
。
相关库不同版本的兼容性并没有详细测过,这里可以给一个我测试用的环境供参考,不一定需要严格符合:
torch == 2.1.0
transformers == 4.39.3
accelerate == 0.29.3
from deepseek-v2.
Related Issues (20)
- 如何优化deepseek用来做文本审查时的prompt定义
- 同一个请求连续多次尝试都是相同错误
- 您好,可以查看源码吗?
- 网页端的默认参数
- 怎么在langchain里面使用deepseek计算embedding? HOT 2
- 希望做vs2022扩展
- Add support llama.cpp
- What's the Prompt and Response length in the Paper?
- 怎么用dspy里的方法来调用deepseek? HOT 2
- Why max_model_len only 8192 when inferencing with vLLM for DeepSeek-V2-Chat?
- about the active param counts of DeepSeek-V2-Lite
- 为何我在A800上运行DeepSeek-V2-Lite-Chat (SFT),竟然消耗60G的显存?! HOT 3
- 0628版本加载报错
- Error executing method determine_num_available_blocks: vLLM multi node fails for both DeepSeek-Coder-V2-Instruct and DeepSeek-Coder-V2-Lite-Instruct
- 自配大模型服务器如何选择GPU,CPU和内存
- DeepSeek-V2-Lite-Chat模型启动依赖问题
- 多轮在训练中是否需要特殊间隔符,用什么间隔符号?
- HuggingFace中开源的代码似乎没有实现矩阵合并
- empty response from server HOT 2
- ValueError: The model's max seq len (163840) is larger than the maximum number of tokens that can be stored in KV cache (13360). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepseek-v2.