terrierteam / pyterrier_ance Goto Github PK

In response to a discussion with Sarawoot.

It would be good to have a reranker transformer for ANCE.
This is possible as IndexFlat support reconstruct() methods. See https://github.com/facebookresearch/faiss/wiki/Faiss-indexes#supported-operations

Why is the args parameter local to the index method of ANCEIndexer instead of being defined as an attribute of the class instance? This makes it difficult to modify the arguments passed to the ANCE module (I would have to subclass the indexer class and implement a new index method which is identical to the one defined here, but with the args I want to pass).

args is defined on the class instance for ANCERetrieval so it probably makes sense to do the same for ANCEIndexer.

If you agree I'll open up a PR with this change.

I am wondering if there is a way to use ANCEIndexer to index my own dataset to enable applying dense retrieval? My dataset does not exist in your Dataset API.
@cmacdonald @seanmacavaney @tonellotto @Xiao0728

From what I can tell, there's currently no way to use ANCE as a plain old re-ranker. While this is not the intention for ANCE in practice, it would be nice to have in some situations.

Would you accept a PR that adds an ANCEReRanker?

I noticed that during indexing my processing time goes up significantly after the first segment has been indexed. Looking at the code I see faiss.omp_set_num_threads(16) after the first segment has been encoded (after the first call to StreamInferenceDoc). Pulling up htop I see that only 16 cores are in fact being utilized. My understanding is that setting the number of OMP threads is a global change, not something that is local to the FAISS index, so the encoding step is also being affected by this.

Is this intentional (meaning, is there a reason to set the number of threads to 16 rather than some other number, and why is it only set after the first segment is encoded)?

If not, I can open a PR to take in the number of threads as an argument (although I would probably expect the calling code to handle setting the number of threads. If this was my project, I would remove the line entirely (as I seen you've done on the retrieval side (it is commented out))).

Thanks!

Hi,

I am trying to use ANCEIndexer to index several datasets at once. I created one iter of dicts for three collections using itertools.chain:

corpus_iter = itertools.chain(
    wapo_generator,
    pt.get_dataset("irds:kilt").get_corpus_iter(verbose=False),
    pt.get_dataset("irds:msmarco-passage").get_corpus_iter(verbose=False),
)

where wapo_generator is my own iterable of dicts that have the keys "docno", "docid" (which is an original id of the document in the collection), and "text". The index is created and I'm able to perform a search. Now, I would like to get the original ids of the retrieved documents (the ones from original collections, e.g. "MARCO_D820886'). Is there any way to do that?

@cmacdonald @seanmacavaney @tonellotto @Xiao0728

Hi,

I'm trying to use ANCE dense retriever followed by a T5 re-ranker implemented using the transformers package. I'm getting the dependency conflicts, as pyterrier_ance requires transformers==2.3.0, but I need to use a newer version that contains T5 model. Currently, I'm using transformers==4.9.0.

@cmacdonald @seanmacavaney @tonellotto @Xiao0728