Comments (9)
Hi Nandan,
Thanks for your interest in our work. I planned to update you on this repo after I released the training code :)
BEIR is a very fascinating benchmark, and it will be great to evaluate RepCONC on it. I will try testing RepCONC on one of the selected datasets (e.g., TREC-COVID) and then share the code. I will update here when the code is ready.
Best,
Jingtao
from repconc.
Hi @jingtaozhan,
Thank you so much! I look forward to the code when it's ready!
Kind Regards,
Nandan Thakur
from repconc.
Hi @jingtaozhan,
I understand that you will be busy with other work with higher priorities.
Would it be possible for you to provide an approximate timeline for this? If possible.
Thanks!
Kind Regards,
Nandan Thakur
from repconc.
Hi @NThakur20
I'm currently working on it and it is almost done. The repo will be updated today. RepCONC will utilize the JPQ package to perform zero-shot retrieval for BEIR. Here is how I evaluate JPQ on BEIR.
I write code by following the model examples in BEIR repo. So I think both JPQ and RepCONC can be added to the BEIR examples. What do you think?
Best,
Jingtao
from repconc.
Awesome, thank you so much @jingtaozhan!
Yes, I believe both will be interesting to evaluate, I'm currently working on a paper where we are evaluating several memory compression strategies. I can add these examples to the BEIR examples folder as well :)
Kind Regards,
Nandan Thakur
from repconc.
The code is released now. Happy to help if you have any other questions.
I do think it's very meaningful to include JPQ and RepCONC in BEIR examples. They follow a different paradigm, joint optimization with compact index, compared with many existing DR models. It wouldn't be hard to add them since the code is ready. I can open a PR if you think is OK.
Best,
Jingtao
from repconc.
Thanks @jingtaozhan, for providing the scores and the script soon.
I think it will be definitely interesting to add the models to the repository. I would be happy if you can open a PR in the repository.
Thanks,
Nandan Thakur
from repconc.
Hi @NThakur20
I update the code so that it is now very easy to apply RepCONC to different dense retrieval models (Pull Request). Thought you might be interested.
Best,
Jingtao
from repconc.
Hi @jingtaozhan,
Thank you for the PR and apologies for the delay. Will have a look.
Meanwhile, in a recent preprint of ours, we found our work on GPL (https://aclanthology.org/2022.naacl-main.168/) useful to help improve zero-shot JPQ performance. We experimented with JPQ on all BEIR datasets (with TAS-B as backbone instead of STAR). GPL which involves cross-encoder distillation with MarginMSE loss function helped improve JPQ models across all BEIR datasets and even outperformed the original uncompressed TAS-B model. For more details, you can have a look below. Would like to get some feedback :)
Paper: https://arxiv.org/abs/2205.11498
Code: https://github.com/thakur-nandan/income
Kind Regards,
Nandan Thakur
from repconc.
Related Issues (5)
- Release of the training code HOT 6
- Fixing of Index Assignments HOT 5
- Unsupervised PQ results HOT 1
- Multi-gpu training HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from repconc.