r0maink / outperformer Goto Github PK

Question
Here again 😄! Seems hard for me to think how to apply attention mask to fast attention, can you please shed some light on that?

I think I should fill some of the Q' and K' to 0 according to the attention_mask, since Q' @ K'.T equals the matrix A, but is that correct?

Question
In the fast_attention.py, I saw

self.register_buffer("orf", create_orf(d_model // h, m), persistent=False)

What puzzled me is that persistent=False line, which says orf won't be in the state dict of this class. I'm wondering, when you finished training, don't you need to remember the last used orf for inferencing?