Question about padding the input sequence about stanford_alpaca

[srhthu]

I think so.. Actually they use the dynamic padding by the "DataCollatorForSupervisedDataset". My concern is should the padding tokens be at left rather than right? The other repo https://github.com/tloen/alpaca-lora padding to the left, which makes sense for batch training.

[maksimstw]

Agree with @srhthu. I think left padding makes more sense, but the train.py used right padding instead. I think the code they use to train Alpaca is simply not correct for batch training. See the explanation here.

[BaleChen]

Hey @maksimstw,

My previous understanding is that batch inference with decoder models requires us to do left padding. But at the fine-tuning stage, right-side padding is okay as long as we set the attention mask correctly and turn pad tokens to -100 when calculating loss.

Is it the case that we can just simply use left padding for both training and inference in generation tasks?