bbuf / flash-rwkv

Hi there, 👋

I'm BBuf, Working At SkyworkAI Now.

My side projects

1. Learning note on CUDA, AIComplier, LLM:

   

   

   

2. Traditional image processing algorithm sharing:

   

   Note: Currently, this project only includes explain blog in Chinese.

3. Deep learning technology sharing:

   

   

4. Algorithm optimization:

   

   

5. Contibutions To Open Source Projects:

   I have made some contributions to some famous open source projects such as pytorch，oneflow, TVM etc.

6. Keep writing and publishing articles about tech comm (technical writing, stories, self-improvement, etc.) in Chinese:

   • 知乎专栏：GiantPandaCV
   • 微信公众号：GiantPandaCV
   • CSDN博客：just_sort

flash-rwkv's Issues

关于 rwkv 的推理速度疑问

你好，我使用 huggingface 的 transformers 库对 huggingfacehub 上面的 llama2-7b和 rwkv-7b进行了下载，然后针对不同的输出长度在 A100-40GB上进行了对比，得到的结果如上图所示。因为 rwkv 在论文中提到了 rwkv 的比较大的优势就是推理速度比较快，但是现在的测试结果看起来比 transformer 类的大模型似乎要慢非常多。请问您和您的团队有测试过这方面的数据吗，或者您和您的团队是否知道原因。我非常乐意继续探索这个图上的结果出现的原因，以及继续进行优化。
如果您能给予一些意见和指导我将不胜感激。

Triton support

If you're planning to make this API somehow standardized it would be great to integrate Songlin Yang's excellent new Triton RWKV-6 implementation from FLA
https://github.com/sustcsonglin/flash-linear-attention/blob/main/fla/ops/rwkv6/chunk.py

Another note is that the RWKV-6 kernel can be used for RWKV-5 with the proper expansion of the w tensor, so its possible to support just that one if desired for simplicity and maintenance.

Also, I don't know if your CUDA version supports true full backprop of the state gradients (none of Bo Peng's do fully yet). I believe hers does.