Problem:
I want to retrieve all relevant (similar) documents from the ElasticsearchDocumentStore
based on the _score
using the EmbeddingRetriever
(I am not using the Reader). Prior to the search, I don't know how many relevant Documents exist. To make sure, that I retrieve all relevant entries from the ElasticsearchDocumentStore
I need to set top_k=10000
or higher and filter the results afterwards - only taking documents with a _score
higher than x. Retrieving this many documents takes several seconds.
Solution
Filtering your query results by a minimum score value is already implemented in the Python Elasticsearch client. You could add another parameter (min_score
) similar to tok_k
and add it to the body that you use in client.search()
. See my example:
body = { "size": top_k, "min_score": min_score, "query": self._get_vector_similarity_query(query_emb, top_k) }
I changed the body form the function def query_by_embedding(...)
from the file haystack/document_stores/elasticsearch.py. Now the results contain only documents that have a _score
higher than min_score
.
Additional context
In case the user wants to filter the results by the cosine similarity metric the min_score parameter needs to be scaled appropriately before using it in the body.