on [PAD] as an actual token about colbert

[okhat]

Hi Craig! I just saw this ECIR'21 preprint that seems to do some analysis about alignment. https://arxiv.org/abs/2012.09650

Not much about [MASK]s though.

[okhat]

That's interesting and, in hindsight, is to be expected with other special tokens in any large collection (e.g., there's probably [SEP] too).

I guess the right thing is to tokenize them into three tokens including the square brackets?

But I don't think this has any significant impact, unless someone is searching for special tokens?

[cmacdonald]

I've been adapting code looking at alignments, and this tripped me up. I wondered if the x!=0 is really needed in colbert.mask(), as special tokens are removed using the normal attention masking?

[okhat]

Surprisingly, it actually matters with HuggingFace. The attention mask doesn't prevent non-zero outputs. It just makes sure other tokens don't attend to the masked positions.

[okhat]

The other thing is that the masks remove pad tokens before saving documents to disk in indexing. This probably saves some space.

[cmacdonald]

Interesting. I was surprised that the BERT tokeniser would accept [PAD] as a token - I though special tokens had to be appended to the generated ids; I guess the impact is for queries involving something abbreviated as PAD, SEP etc.

Sure, trailing PADs need to continue to be removed. Anyway, its an observation rather than a bug.

[okhat]

I agree it's worth addressing, especially in a production system. The easiest way is to surround all square brackets with spaces in the original text (i.e., via an all replace).