[BUG] model的forward函数接收attention_mask的时候，若attention_mask[i, 0]==0，则序列i输出的logits全都是NaN值 about qwen

[jklj077]

Hi, I'm not sure I understand your use case. May I know what results you were expecting? You have literally prevented the initial position from attending to itself and it should be expected that the model did not know what next token would be.

[leileqiTHU]

yeah sorry that I may not make it clear.

I was trying to use model.forward function directly rather than calling model.generate function, in order to observe its behavior in the forward pass.
My input is of different lengths, so I have to pad them to the same lengths. I used left padding, prepending pad <endoftext> tokens. In my opinion, those pad tokens should not be attended, and attention_mask is used in this scenario, setting those positions to 0 so that the model won't attend to those pad tokens in the forward pass.
However, I got all NaN logits, which confuses me. I tried not to pass the attention_mask parameter, and there are no NaN values in the logits, which is I expected. So I infer that this may be the problem of the attention_mask. To further locate the problem, I tried different attention_masks, finally found out that If we set the first position to 0 (in which case the model won't attend to the first token which is a pad token), the return values of model.forward function , i.e., the logits, will all be NaN values.

Also, I tried Qwen1.5-7B-Chat model, and it does not have this problem, i.e., even if I set the attention_mask of the first position to 0, the output will still be free of NaN values. So I suspect that this may be a problem of Qwen-7B-Chat.

But also, I may make mistakes, please let me know if I do.

[leileqiTHU]

And If the masked tokens in the left positions should not know what the next token should be due to that they are prevented from attending to themselves, why are the logits of other un-masked positions (the right positions ) are also NaN values? Did I get it wrong?

[jklj077]

Hi, after reading through your comments, and if I understood correctly, Qwen1.5 was working as you would expect. I would suggest just using Qwen1.5.

P.S.: Investigating the original issue is more complicated than it appeared. Was flash attention enabled? Were you following the instructions in README to do batch inference?

[cageyoko]

遇到了相同的问题，

1. 不用flash-attn batchsize=1可以正常出结果 batchsize>1时候有padding的样本过模型后输出为nan
2. 安装flash-attn 就好了

[jklj077]

Hi, Qwen1.0 models and code will not be updated anymore. Please try Qwen2.0 instead.