openmatch / coco-dr

[EMNLP 2022] This is the code repo for our EMNLP‘22 paper "COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning".

OpenMatch v2

An all-in-one toolkit for information retrieval. Under active development.

Install

git clone https://github.com/OpenMatch/OpenMatch.git
cd OpenMatch
pip install -e .

-e means editable, i.e. you can change the code directly in your directory.

We do not include all the requirements in the package. You may need to manually install torch, tensorboard.

You may also need faiss for dense retrieval. You can install either faiss-cpu or faiss-gpu, according to your enviroment. Note that if you want to perform search on GPUs, you need to install the version of faiss-gpu compatible with your CUDA. In some cases (usually CUDA >= 11.0) pip installs a wrong version. If you encounter errors during search on GPUs, you may try installing it from conda.

Features

• Human-friendly interface for dense retriever and re-ranker training and testing
• Various PLMs supported (BERT, RoBERTa, T5...)
• Native support for common IR & QA Datasets (MS MARCO, NQ, KILT, BEIR, ...)
• Deep integration with Huggingface Transformers and Datasets
• Efficient training and inference via stream-style data loading

Docs

We are actively working on the docs.

Project Organizers

• Zhiyuan Liu
  • Tsinghua University
  • Homepage
• Zhenghao Liu
  • Northeastern University
  • Homepage
• Chenyan Xiong
  • Microsoft Research AI
  • Homepage
• Maosong Sun
  • Tsinghua University
  • Homepage

Acknowledgments

Our implementation uses Tevatron as the starting point. We thank its authors for their contributions.

Contact

Please email to [email protected].

coco-dr's Issues

Something questions about part'Pre-processing'

Hello, thanks for your interesting work!

I'm tring to recomplete COCO Pre-training and I noticed that I need to preprocess the dataset.
This is mentioned in the ./COCO-DR/COCO/README.md

But when I follow the instructions in it, Something goes wrong in pre_processing_coco.sh.
It calls COCO-DR/COCO/helper/create_train_co_short.py and there's a function called encode_one().

in the line 35&36, item is a Dict but no group, spans key in the Dict. This will cause raise valueKeyError: 'group'

log as follows:

I noticed that there are only four keys in each line of the dataset: 'id','title',"text','metadata'
Did I miss some steps before preprocessing?
I'm eagerly looking forward to your reply!!! Thanks a lot!

Best regards!

Cannot find Command folder for ACNE

Hello,

I find this project very interesting.

I cannot see any folder for commands/install.sh and commands/run_ance.sh

Could you please upload these files?

Best regards,
Markus

Questions about the results in Table 7 in the paper

Hi, I am very interested in your work and have some small questions.
As the results in Table 7 in the paper are not explained too much, I want to know is the metric MRR?
And did you evaluate the results at CodeSearchNet evaluation dataset with 99 queries?
By the way, is there any way to reproduce the results in Table 7 with the code in this repo?

Thanks a lot!

Codes for inference/evaluation

Thank you for the interesting work!

I am wondering if codes for the inference/evaluation on BEIR datasets are available or planned to be shared.

I tried using open-sourced BEIR evaluation codes as follows.
I evaluated COCO-DR (Base) on SciFact dataset.

#### Download scifact.zip dataset and unzip the dataset
dataset = "scifact"
url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{}.zip".format(dataset)
dataset_dir = 'experiments/datasets/beir/'
data_path = util.download_and_unzip(url, dataset_dir)
#### Provide the data_path where scifact has been downloaded and unzipped
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")

#### Load the SBERT model and retrieve using cosine-similarity
model = DRES(models.SentenceBERT("OpenMatch/cocodr-base-msmarco"), batch_size=16)
retriever = EvaluateRetrieval(model, score_function="dot")
results = retriever.retrieve(corpus, queries)

#### Evaluate your model with NDCG@k, MAP@K, Recall@K and Precision@K  where k = [1,3,5,10,100,1000] 
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)

But I got a lower performance than the reported performance in the paper: I got 0.254 nDCG@10 (reported number is 0.709).

2022-11-08 11:37:22 - NDCG@1: 0.1467
2022-11-08 11:37:22 - NDCG@3: 0.2189
2022-11-08 11:37:22 - NDCG@5: 0.2352
2022-11-08 11:37:22 - NDCG@10: 0.2535
2022-11-08 11:37:22 - NDCG@100: 0.3066
2022-11-08 11:37:22 - NDCG@1000: 0.3415

Best regards,
Jihyuk