Checkpoints + usage of different base BERT model about colbert

[littlewine]

1. You can find a similar version of the model trained by other people here. tbh I am not sure how you can this huggingface pretrained model in a downstream task (ie. index marco or doing retrieval) - in other words whether this is compatible with the existing code repo. It is also a question for me whether we can directly use huggingface models instead of the expected colbert checkpoints that this repo expects during inference (either for retrieval or indexing) @okhat could you help us on that? :)
2. In this case, you need to train from scratch and make the necessary changes in the code. Briefly, you'd need to go through all of those code matches and change the input of from_pretrained to the model string you want. Surely in the training.py script but also possibly in evaluation

[zyngielg]

@littlewine thanks for the reply

@1 I found this model, however it has slightly different parameters (linear layer output size and from what I remember the lenght of the embeddings being 512 instead of 768). Still, for now I indeed plan to use that one :) And about the usage in the downstream task - I found it slightly difficult to follow the retrieve script's logic and I guess the smartest choice would be to follow the description of the workflow in the paper.

[okhat]

Hey folks! Happy to send you a checkpoint if you send me an email. A public/official release will take some time.

There are public checkpoints by other people that are compatible with (and produced from) this code, including that by PyTerrier-ColBERT: https://github.com/terrierteam/pyterrier_colbert --- see the Demo section.

[jobergum]

We over at vespa.ai have released a ColBERT weights which is based on a 6 layer MiniLM model with just 22.7M parameters which uses 32 dimensions instead of 128. So for question 2 you can replace bert-base-uncased with a different model which also uses BERT tokenization.

You can use https://huggingface.co/vespa-engine/col-minilm and then use the save_checkpoint to get a .dnn checkpoint which can be used with more routines (but with similarity=cosine and dim=32).
(

ColBERT/colbert/utils/utils.py

Line 37 in abb5b68

def save_checkpoint(path, epoch_idx, mb_idx, model, optimizer, arguments=None):

)

[zyngielg]

Thanks all for your activity in the discussion :)

[shunyuzh]

Hi @zyngielg,
Now do you have any idea about how to use huggingface API to create the ColBERT model in the code and load the checkpoints?