Can't build faiss index about colbert

[okhat]

This command seems like the initial indexing step, not the faiss step. Is that right?

I would run -m colbert.index from outside the colbert/ subdirectory to make sure the imports work. Also, don't pass local_rank. Basically, please follow the template in the README.

Let me know if issues still persist!

[okhat]

Another orthogonal thing is: do you want to index queries.tsv? Typically you'd want to index passages.

[JamesDeAntonis]

yes, they're passages. The file is under a different name and i mistakenly put queries.tsv instead of passages.tsv

[JamesDeAntonis]

unfortunately, they still persist

what do you mean by initial indexing step?

[okhat]

I'm assuming this is the first step of indexing (before colbert.index_faiss is used).

If so, here's a sample script that's modified based on your command above:

CUDA_VISIBLE_DEVICES="0,1" OMP_NUM_THREADS=1 \
python -m torch.distributed.launch --nproc_per_node=2 -m \
colbert.index --root $PWD/experiments/ --amp --doc_maxlen 180 --mask-punctuation --bsize 256 \
--checkpoint out-of-the-box-model.pt \
--collection passages.tsv \
--index_root /faiss --index_name INDEX \
--root $PWD/experiments/ --experiment out_of_the_box \

Just to be sure: does this work for you?

[JamesDeAntonis]

(this the full error, btw)

[JamesDeAntonis]

oh, and yes it's the first step (encoding)

[JamesDeAntonis]

I get this now (slightly different and may be a step in the right direction)

[JamesDeAntonis]

I get this now (slightly different and may be a step in the right direction)

[okhat]

Hmm this is strange. Could you run with one GPU in distributed mode and see if that works?

CUDA_VISIBLE_DEVICES="0" OMP_NUM_THREADS=1 \
python -m torch.distributed.launch --nproc_per_node=1 -m \
colbert.index --root $PWD/experiments/ --amp --doc_maxlen 180 --mask-punctuation --bsize 256 \
--checkpoint out-of-the-box-model.pt \
--collection passages.tsv \
--index_root /faiss --index_name INDEX \
--root $PWD/experiments/ --experiment out_of_the_box \

[JamesDeAntonis]

it didn't :/

[okhat]

Okay. What about single-GPU mode?

CUDA_VISIBLE_DEVICES="0" python -m \
colbert.index --root $PWD/experiments/ --amp --doc_maxlen 180 --mask-punctuation --bsize 256 \
--checkpoint out-of-the-box-model.pt \
--collection passages.tsv \
--index_root /faiss --index_name INDEX \
--root $PWD/experiments/ --experiment out_of_the_box \

[JamesDeAntonis]

It's hanging right now, so might be working!

[JamesDeAntonis]

ah yes, getting another error that I have already seen, and then it continues to hang (i saw this yesterday):

[okhat]

Right, so your passages must be in a TSV file whose first column is the passage ID.

Passage ID must be equal to the line number, starting at 0. You can start at 1 also, but then line zero needs to have a header: id \t text

[JamesDeAntonis]

Oh, mine start at 1. first line is 1\tsome passage

[JamesDeAntonis]

Great to know! I'll fix that

[okhat]

Okay, so single-GPU encoding will work.

For debugging the multi-GPU setup, could you run the following in a Python terminal?

import torch
torch.cuda.device_count()

[JamesDeAntonis]

1

[okhat]

Ah, so this all makes sense then. Either you only have one GPU or pytorch is not able to detect your other GPUs.

[JamesDeAntonis]

(I'm not sure why that is, by the way. I have two GPUs)

[JamesDeAntonis]

But yeah that makes sense, I'll look into that. Thank you so much for your prompt replies! Very helpful

[okhat]

Looks resolved?

Closing but feel free to reopen if I can help with other issues!