Comments (10)
@Copilot-X 请问你的运行代码是怎么样的呢?理论上用 bf16/fp16 加载模型只需要 12GB 左右显存
from yi.
我跑了 demo , 加载了模型后 13G 左右显存占用, 推理时候再多 500MB 左右
from yi.
我跑了 demo , 加载了模型后 13G 左右显存占用, 推理时候再多 500MB 左右
加载推理的代码有么? 我对比一下看看
from yi.
我跑了 demo , 加载了模型后 13G 左右显存占用, 推理时候再多 500MB 左右
加载推理的代码有么? 我对比一下看看
就仓库的呀: https://github.com/01-ai/Yi/blob/main/demo/text_generation.py
from yi.
目前模型是用bfloat16数据类型,6B模型至少需要13GB左右的显存。
from yi.
200k上下文的6B与34B模型分别需要多少显存?
from yi.
在
Yi\demo\text_generation.py 文件中
加两个参数(需要安装一些依赖库,没安装会报错)后,4G 显存也能跑,但是速度超级慢。
还是要依赖llama.cpp 这种优化方案,否则小显存设备基本没法玩
ChatGLM3 6B 也是使用chatglm.cpp 量化到4 后,才跑的飞起,使用官方量化方案,也基本十几分钟才有回复。
from yi.
想问下,推理速度有多少tokens / s
from yi.
本次 Chat 版本的发布特地增加了该部分内容。
from yi.
按照readme给的代码,用的6B chat 11GB模型,8G显存,显卡是3070Ti
能跑但是很慢很慢,10分钟多了
但是同样的机器我跑chatglm3-6b 也是11GB的模型很快呀,几秒钟就开始输出了,一两分钟就输出完了,
难道是因为这个是一次性输出的?
from yi.
Related Issues (20)
- 偶发性的会报错
- v100显卡,加载量化模型Yi-34B-Chat-4bits,推理速度很慢 HOT 7
- Features : openai_api.py support multi turn dialogs. HOT 1
- Result of Yi-6B-Chat on the BBH dataset cannot be reproduced HOT 1
- Yi-VL-34b支持int4量化吗?怎么操作 HOT 2
- 自定义数据train.jsonl 8万多,eval.jsonl 105条,为什么SFT时候只显示 length of train dataset:2852,length of eval dataset: 9 HOT 1
- When the API is called multiple times, the GPU memory continuously increases until it overflows. HOT 1
- LLama3发表了,啥时候Yi出新版本啊 HOT 2
- RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'” HOT 4
- Test issue bot
- Test issue bot
- where can I find the training code or script for YI-VL HOT 1
- lora微调yi-6b-chat之后,生成的结果会出现大量的换行符以及空格 HOT 4
- YI:9b在长上下下回答异常 HOT 5
- 用自己的数据集微调时会出现下面的报错,但是用官方的yi_example数据集就不会出现报错,请问这是为什么? HOT 1
- 请问有Yi-VL可以实现few-shot(in-context)数据的推理或微调吗? HOT 1
- Let's Build Yi Cookbook Together - Your Ideas Matter! HOT 4
- 拉了一个多模态大模型技术交流群,大家可以加入进来进行技术交流
- 📝 Yi 周边设计集思广益 HOT 1
- 🧠 Yi Merchandise Design Brainstorming!!! 🚀
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yi.