Comments (8)
This is very unusual, it should not return repeated passages unless there’s some recent change causing that.
do you have a guess @santhnm2 @detaos @jessiejuachon
from colbert.
Agree with @okhat, this is very strange - I'll look into it.
from colbert.
I'm running into another issue with large k
values.
Running search with any large k
, e.g. 100, returns exactly 64 passages.
from colbert.
@s-jse I've experienced that issue as well. The number of passages returned seems to get stuck at a specific number (a power of 2) no matter what value is passed. This was noticed to have happened whenever we vary the k value in a for-loop and it freezes at a specific number beyond which it doesn't return results even if they exist. Code snippet used to test this:
def get_answers(query):
for k in [1, 10, 100]:
print(f"Question: {query} k: {k}")
results = searcher.search(query, k=k)
print(f"NUM RESULTS: {len(results[0])}")
unique_pids = set()
for passage_id, passage_rank, passage_score in zip(*results):
print(f"passage_id={passage_id} passage_rank={passage_rank} passage_score={passage_score}")
unique_pids.add(passage_id)
print(f"UNIQUE PIDS: {len(unique_pids)}")
sleep(2)
if __name__ == '__main__':
searcher, output_indexes, output_content = load_model()
narratives = load_dataset()
for i in range(100):
narrative_dict = narratives[i]
get_answers(narrative_dict['question'])
from colbert.
I used Colbert for quite a lot those past days and I fixed this one for myself.
So I wanted to share a solution. I submitted a PR.
It's my first time submitting to an open source project and welcome any feedbacks.
Cheers
from colbert.
@paul7Junior really appreciate this! We’ll take a look!
from colbert.
Hello! I tried to search using a large
k
value. I noticed that ColBERT returns some unique results, but when there aren't enough results, it soon starts returning duplicate results (with the same passage ID) until it reaches thek
value specified.For example, the code below:
query = "some question" # or supply your own query print(f"Question: {query}") results = searcher.search(query, k=1000) print(f"NUM RESULTS {len(results[0])}") unique_pids = set() repeated_pids = [] for passage_id, passage_rank, passage_score in zip(*results): text = searcher.collection[passage_id] if passage_id in unique_pids: repeated_pids.append(passage_id) continue unique_pids.add(passage_id) print(len(repeated_pids), repeated_pids) print(len(unique_pids), unique_pids)
prints the following:
Question: some question NUM RESULTS 1000 872 [25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, ...] # seems to duplicate just one passage over and over again 128 {1, 1027, 1034, 2572, 1041, 1043, ...}
Shouldn't ColBERT only be returning unique results? Is this is a known bug?
User code below, I was facing the same Issue, but using this custom_config object I was able to get 100 results or more.
custom_config = ColBERTConfig(ncells=1000, ndocs=1000,reranker=True)
with Run().context(RunConfig(experiment='notebook')):
searcher = Searcher(index=index_name,config=custom_config)
from colbert.
Hey Ravi Kumar,
Thanks for your comment,
Can you just post here the version of colbert-ai that you're using.
Also, when you say you was able to get 100+ results, are you talking about 100+ results of the same passage id?
from colbert.
Related Issues (20)
- CollectionEncoder blocking on encoder N passages HOT 1
- Focusing retrieval on list of document ids with doc_ids parameter doesn't work
- type object 'ColBERT' has no attribute 'segmented_maxsim' HOT 1
- Where is the qrels.dev.small.tsv?
- How to get rid of the "Duplicate GPU detected : rank 0 and rank 1 both on CUDA device ca000" error while training of the ColBERTv1.9 modell? HOT 1
- Request for AMD gpu support
- How to quickly check if installation is working fine?
- ColBert is not failing when Error is encounter during both train and indexing
- How to insert new document into the pre-built index? HOT 1
- Is there a check point of ColBERT that wasn't trained on MSMARCO?
- How to check the centroids and the data in the clusters?
- Extract only embeddings
- Execution fails in colbert.index_objs() with assert classname.endswith('Vector')
- Results on BEIR
- unable to open file </root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/86b5e0934494bd15c9632b12f734a8a67f723594/model.safetensors> in read-only mode: No such file or directory (2)
- Add_to_index only work first time
- Tokenization Assumption for Query Marker Replacement is Inconsistent
- GPU crashes when running "D_packed @ Q.to(dtype=D_packed.dtype).T" with no error message HOT 1
- Training script from doc is not working
- ImportError: cannot import name 'packaging' from 'pkg_resources' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from colbert.