sooftware / attentions

PyTorch implementation of some attentions for Deep Learning Researchers.

An Apache 2.0 PyTorch implementation of some attentions for Deep Learning Researchers.

---

Intro

attentions provides some attentions used in natural language processing using pytorch.
these attentions can used in neural machine translation, speech recognition, image captioning etc...

attention allows to attend to different parts of the source sentence at each step of the output generation.
Instead of encoding the input sequence into a single fixed context vector, we let the model learn how to generate a context vector for each output time step.

Implementation list

Name	Citation
Additive Attention	Bahdanau et al., 2015
Dot-Product Attention	Luong et al., 2015
Location-Aware (Location Sensitive) Attention	Chorowski et al., 2015
Scaled Dot-Product Attention	Vaswani et al., 2017
Multi-Head Attention	Vaswani et al., 2017
Relative Multi-Head Self Attention	ZihangDai et al., 2019

Troubleshoots and Contributing

If you have any questions, bug reports, and feature requests, please open an issue on Github.
or Contacts [email protected] please.

I appreciate any kind of feedback or contribution. Feel free to proceed with small issues like bug fixes, documentation improvement. For major contributions and new features, please discuss with the collaborators in corresponding issues.

Code Style

I follow PEP-8 for code style. Especially the style of docstrings is important to generate documentation.

Author

• Soohwan Kim @sooftware
• Contacts: [email protected]

attentions's Issues

sharpening feature

Attention-Based Models for Speech Recognition has a sharpening feature which selects a few of hidden states to attend. However, there are little details about how to select in the paper and your code.

something about the additive attention

Hello, I think If you want the additive attention be able to deal with batch, while inputs are like these
Inputs: query, value
- query (batch_size, q_len, hidden_dim): tensor containing the output features from the decoder.
- value (batch_size, v_len, hidden_dim): tensor containing features of the encoded input sequence

the code in forward function should be like this:
def forward(self, query: Tensor, key: Tensor, value: Tensor):
score = self.score_proj(
torch.tanh(self.key_proj(key.unsqueeze(1)) + self.query_proj(query.unsqueeze(2)) + self.bias)).squeeze()
attn = F.softmax(score, dim=-1)
context = torch.bmm(attn, value)
return context, attn

otherwise, the size of self.key_proj(key.unsqueeze(1)) and self.query_proj(query.unsqueeze(2) will be dismatch on second dimension and can not be added

Unused self.out_projection

In 2 places:

attentions/attentions.py

Line 52 in d6a6b12

self.out_projection = nn.Linear(hidden_dim * 2, hidden_dim)

attentions/attentions.py

Line 192 in d6a6b12

self.out_projection = nn.Linear(hidden_dim << 1, hidden_dim, bias=True)