Comments (4)
The same issue was reported in #38 (comment) by @burgerlawful
UPD: same behaviour happens on 0.7.2
from chatrwkv.
@nenkoru pls try latest ChatRWKV & rwkv 0.7.3
should be fixed now
from chatrwkv.
Yeah, it works now. But I don't see any big difference between turning this feature on or off. Here are timings per .forward call. However, I noticed slightly less memory usage of around 1.5 - 2GB for 14B model so I managed to fit a little more layers without quantization into i8. It's also worth mentioning that the CPU on the machine fairly slow - AMD Athlon 3000G with 2 cores(4 threads) that could be the bottleneck in my case.
Cuda optimization OFF. 7B
cuda:0 fp16 -> cuda:1 fp16 -> cuda:2 fp16
forward time: 0.023224592208862305
forward time: 0.02307891845703125
forward time: 0.023171663284301758
forward time: 0.023991107940673828
Cuda optimization ON. 7B
cuda:0 fp16 -> cuda:1 fp16 -> cuda:2 fp16
forward time: 0.023212432861328125
forward time: 0.021632671356201172
forward time: 0.021616697311401367
forward time: 0.022366046905517578
Cuda optimization ON. 14B.
cuda:0 fp16 *10 -> cuda:1 fp16 *10 -> cuda:2 fp16 *10 -> cuda:3 fp16 *10 -> cuda:3 fp16i8
forward time: 0.02741098403930664
forward time: 0.030609130859375
forward time: 0.029874563217163086
forward time: 0.02969217300415039
forward time: 0.028966665267944336
forward time: 0.030252695083618164
Cuda optimization OFF. 14B.
cuda:0 fp16 *10 -> cuda:1 fp16 *10 -> cuda:2 fp16 *10 -> cuda:3 fp16 *7 -> cuda:3 fp16i8
forward time: 0.029529571533203125
forward time: 0.030321836471557617
forward time: 0.02967238426208496
forward time: 0.030389785766601562
forward time: 0.03206944465637207
forward time: 0.030094385147094727
from chatrwkv.
@nenkoru It's 10x faster for long inputs
Use more fp16i8 and 2 GPUs will be enough.
Use fp16 in later (instead of early) layers for better quality.
from chatrwkv.
Related Issues (20)
- huggingface无法访问,模型无法下载 HOT 4
- Prompt for RAG with RWKV-4-World-7B-v1-20230626-ctx4096 HOT 1
- [Feature Request] text2music HOT 2
- RuntimeError: Error building extension 'wkv_cuda_v1' HOT 2
- How to write the RWKV in autogressive style like RNN HOT 2
- NameError: name 'PIPELINE' is not defined HOT 1
- 大哥,乱码了 HOT 1
- 回复总是截断了,如何让回复自然的结束 HOT 1
- eagle-7B HOT 1
- Inference doesn't work on Apple Macbook even when using CPU fp32 as strategy HOT 1
- "cpu fp32i8" strategy not working in RWKV v6 through Python rwkv module HOT 2
- How to run new v5-Eagle-7B HOT 2
- mps slower than cpu HOT 1
- model path list HOT 1
- add text condition for gen music HOT 1
- [pip package] Make loading aware that os.environ can change HOT 2
- [pip package] feature request: pipeline.generate: add ability to get the state, if it was not provided HOT 1
- 如何选模型基座? HOT 1
- [requires_grad]在本地部署CHATRWKV时遇到了AttributeError: 'str' object has no attribute 'requires_grad' HOT 1
- 加速模型载入 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chatrwkv.