Comments (2)
By using scaled_dot_product_attention
, model.py
is using Flash Attention 2, as scaled_dot_product_attention
uses Flash Attention 2 by default now (pytorch/pytorch#105602)
from gpt-fast.
Yes what @MF-FOOM said. In addition, FA2 is not beneficial during the decoding phase, so it's actually more beneficial for us to codegen it from scratch. It would be faster to use the FlashDecode kernel, but that's not fully integrated into PyTorch yet.
from gpt-fast.
Related Issues (20)
- int8 Woq raise Codegen Error with `--compile_prefill` HOT 4
- On the memory usage of `ConditionalFeedForward` HOT 4
- Tiny Llamas Not Found HOT 2
- INT4 quantization not working on MI210 HOT 2
- Input token length question HOT 2
- Naming: n_local_heads -> n_kv_heads
- mmap issue in bf16 of gpt-fast
- Tensor Parallel Inside notebook
- Missing Keys in state_dict HOT 1
- Throughput Benchmark Scripts HOT 2
- Hard-coded Llama-3 model name pattern matching breaks scripts/convert_hf_checkpoint.py
- `meta-llama/Meta-Llama-3-8B-Instruct` generates gibberish for long prompts HOT 5
- GGUF support?
- Question about the ENABLE_INTRA_NODE_COMM for speculative decoding HOT 9
- getting different acceptance prob when using `torch.compile` after making a small change.
- It doesn't accelerate very well at L4
- tokenizer.model
- Support of FlashDecoding HOT 3
- trying to convert huggingface whisper model to pytorch
- permute function in `convert_hf_checkpoint.py` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpt-fast.