openmatch / coco-dr Goto Github PK

Hello,

I find this project very interesting.

I cannot see any folder for commands/install.sh and commands/run_ance.sh

Could you please upload these files?

Best regards,
Markus

Hi, I am very interested in your work and have some small questions.
As the results in Table 7 in the paper are not explained too much, I want to know is the metric MRR?
And did you evaluate the results at CodeSearchNet evaluation dataset with 99 queries?
By the way, is there any way to reproduce the results in Table 7 with the code in this repo?

Thanks a lot!

Thank you for the interesting work!

I am wondering if codes for the inference/evaluation on BEIR datasets are available or planned to be shared.

I tried using open-sourced BEIR evaluation codes as follows.
I evaluated COCO-DR (Base) on SciFact dataset.

#### Download scifact.zip dataset and unzip the dataset
dataset = "scifact"
url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{}.zip".format(dataset)
dataset_dir = 'experiments/datasets/beir/'
data_path = util.download_and_unzip(url, dataset_dir)
#### Provide the data_path where scifact has been downloaded and unzipped
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")

#### Load the SBERT model and retrieve using cosine-similarity
model = DRES(models.SentenceBERT("OpenMatch/cocodr-base-msmarco"), batch_size=16)
retriever = EvaluateRetrieval(model, score_function="dot")
results = retriever.retrieve(corpus, queries)

#### Evaluate your model with NDCG@k, MAP@K, Recall@K and Precision@K  where k = [1,3,5,10,100,1000] 
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)

But I got a lower performance than the reported performance in the paper: I got 0.254 nDCG@10 (reported number is 0.709).

2022-11-08 11:37:22 - NDCG@1: 0.1467
2022-11-08 11:37:22 - NDCG@3: 0.2189
2022-11-08 11:37:22 - NDCG@5: 0.2352
2022-11-08 11:37:22 - NDCG@10: 0.2535
2022-11-08 11:37:22 - NDCG@100: 0.3066
2022-11-08 11:37:22 - NDCG@1000: 0.3415

Best regards,
Jihyuk

Hello, thanks for your interesting work!

I'm tring to recomplete COCO Pre-training and I noticed that I need to preprocess the dataset.
This is mentioned in the ./COCO-DR/COCO/README.md

But when I follow the instructions in it, Something goes wrong in pre_processing_coco.sh.
It calls COCO-DR/COCO/helper/create_train_co_short.py and there's a function called encode_one().

in the line 35&36, item is a Dict but no group, spans key in the Dict. This will cause raise valueKeyError: 'group'

log as follows:

I noticed that there are only four keys in each line of the dataset: 'id','title',"text','metadata'
Did I miss some steps before preprocessing?
I'm eagerly looking forward to your reply!!! Thanks a lot!

Best regards!