cvangysel / pytrec_eval Goto Github PK
View Code? Open in Web Editor NEWpytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.
Home Page: http://ilps.science.uva.nl/
License: MIT License
pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.
Home Page: http://ilps.science.uva.nl/
License: MIT License
As identified by @grodino in terrierteam/ir_measures#42
In short, when a measure with multiple cutoffs are provided to RelevanceEvaluator
, only one is returned on subsequent invocations.
>>> import pytrec_eval
>>> qrel = {
>>> '0': {'D0': 0, 'D1': 1, 'D2': 1, 'D3': 1, 'D4': 0},
>>> '1': {'D0': 1, 'D3': 2, 'D5': 2}
>>> }
>>> run = {
>>> '0': {'D0': 0.8, 'D2': 0.7, 'D1': 0.3, 'D3': 0.4, 'D4': 0.1},
>>> '1': {'D1': 0.8, 'D3': 0.7, 'D4': 0.3, 'D2': 0.4, 'D10': 8.}
>>> }
>>> evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg_cut.10,100,2'})
>>> print(evaluator.evaluate(run))
{'0': {'map': 0.6388888888888888, 'ndcg_cut_2': 0.38685280723454163, 'ndcg_cut_10': 0.7328286204777911, 'ndcg_cut_100': 0.7328286204777911}, '1': {'map': 0.1111111111111111, 'ndcg_cut_2': 0.0, 'ndcg_cut_10': 0.26582598262939583, 'ndcg_cut_100': 0.26582598262939583}}
>>> print(evaluator.evaluate(run))
{'0': {'map': 0.6388888888888888, 'ndcg_cut_10': 0.7328286204777911}, '1': {'map': 0.1111111111111111, 'ndcg_cut_10': 0.26582598262939583}}
# ^ second invocation is missing ndcg_cut_2, ndcg_cut_100
When trying to install pytrec-eval-terrier=0.5.1, I get the below error:
Collecting pytrec-eval-terrier==0.5.1
Using cached pytrec_eval-terrier-0.5.1.tar.gz (16 kB)
WARNING: Generating metadata for package pytrec-eval-terrier produced metadata for project name pytrec-eval. Fix your #egg=pytrec-eval-terrier fragments.
WARNING: Discarding https://files.pythonhosted.org/packages/6a/38/d723c26698e517f450ea905f633474e8ba714e23fb67a8c2aa34b803efbf/pytrec_eval-terrier-0.5.1.tar.gz#sha256=55e7f5b2c83f681ac262c975f64fdcbece64fed35508d410e56d2c52f262ebfe (from https://pypi.org/simple/pytrec-eval-terrier/) (requires-python:>=3). Requested pytrec-eval from https://files.pythonhosted.org/packages/6a/38/d723c26698e517f450ea905f633474e8ba714e23fb67a8c2aa34b803efbf/pytrec_eval-terrier-0.5.1.tar.gz#sha256=55e7f5b2c83f681ac262c975f64fdcbece64fed35508d410e56d2c52f262ebfe (from ir-measures==0.1.3) has inconsistent name: filename has 'pytrec-eval-terrier', but metadata has 'pytrec-eval'
ERROR: Could not find a version that satisfies the requirement pytrec-eval-terrier==0.5.1 (from ir-measures) (from versions: 0.5.1)
ERROR: No matching distribution found for pytrec-eval-terrier==0.5.1
Seems like there are inconsistent names (perhaps in setup.py
): pytrec-eval-terrier and pytrec-eval
How can I resolve this?
Hi there,
I am getting following error when I try to install pytrec_eval either using pip install or when trying to install python-terrier:
Collecting pytrec_eval
Using cached pytrec_eval-0.4.tar.gz (11 kB)
ERROR: Command errored out with exit status 1:
command: /Users/ali.vahid/project_pyTerrier/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/pb/b4bbdkcn11x0b_2nsvl7kzjh0000gn/T/pip-install-os9v0s7h/pytrec-eval/setup.py'"'"'; file='"'"'/private/var/folders/pb/b4bbdkcn11x0b_2nsvl7kzjh0000gn/T/pip-install-os9v0s7h/pytrec-eval/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/pb/b4bbdkcn11x0b_2nsvl7kzjh0000gn/T/pip-install-os9v0s7h/pytrec-eval/pip-egg-info
cwd: /private/var/folders/pb/b4bbdkcn11x0b_2nsvl7kzjh0000gn/T/pip-install-os9v0s7h/pytrec-eval/
Complete output (43 lines):
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1319, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1004, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 944, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1399, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/pb/b4bbdkcn11x0b_2nsvl7kzjh0000gn/T/pip-install-os9v0s7h/pytrec-eval/setup.py", line 28, in <module>
response = urllib.request.urlopen(REMOTE_TREC_EVAL_ZIP)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1362, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1322, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>
Fetching trec_eval from https://github.com/usnistgov/trec_eval/archive/v9.0.5.zip.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Hi,
Is there any way to specify custom k
values for map_cut
, ndcg_cut
etc., instead of default ones ?
It is definitely possible with trec_eval
, but I don't see how to do it with the current interface (or do I miss something ?)
Thanks
Thibault
The License in pytrec_eval
is MIT but the repo includes https://github.com/usnistgov/trec_eval/tree/de6a29f8ba9312c73f978aa9739695aa8ebf48eb that is under the copyright of Chris Buckley
see also https://github.com/usnistgov/trec_eval/blob/master/convert_zscores.c#L2-L5
and usnistgov/trec_eval#10
I am using
evaluator = pytrec_eval.RelevanceEvaluator(qrels, {map_string, ndcg_string, recall_string, precision_string})
scores = evaluator.evaluate(results)
Now result is
{qid: {docid: embeddding_score, docid: embeddding_score} }
so we sort the results using scores before using for metrics calculation? I am assuming it has to be as dict will not have fix indexing.
can we avoid this sorting? in that case do I have to manipulate the scores manually to achieve desired results?
For some metrics like nDCG, it is plausible that we have float relevance scores.
Is it a way to use pytrec_eval for floating relevance scores?
The following sample:
import pytrec_eval
import json
qrel = {
'q1': {
'd1': 0.2,
'd2': 1.5,
'd3': 0,
},
'q2': {
'd2': 2.5,
'd3': 1,
},
}
run = {
'q1': {
'd1': 1.0,
'd2': 0.0,
'd3': 1.5,
},
'q2': {
'd1': 1.5,
'd2': 0.2,
'd3': 0.5,
}
}
evaluator = pytrec_eval.RelevanceEvaluator(
qrel, {'ndcg'})
print(json.dumps(evaluator.evaluate(run), indent=1))
Raised the following exception:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-7-9cc469855e77> in <module>
28
29 evaluator = pytrec_eval.RelevanceEvaluator(
---> 30 qrel, {'ndcg'})
31
32 print(json.dumps(evaluator.evaluate(run), indent=1))
TypeError: Expected relevance to be integer.
Hello,
I got a series of errors like this:
File "C:\Users\myusername\AppData\Local\Continuum\anaconda3\lib\distutils\util.py", line 111, in convert_path raise ValueError("path '%s' cannot be absolute" % pathname) ValueError: path '/Users/cvangysel/Projects/pytrec_eval/trec_eval/convert_zscores.c' cannot be absolute
when I ran
pip install pytrec-eval
It seems that the absolute path is hardcoded somewhere.
I modified the util.py file to ignore the prefix '/Users/cvangysel/Projects/pytrec_eval/trec_eval/' and the installation seems to be successful (the example on README.md ran correctly).
This works
evaluator.evaluate({'151': {'clueweb09-en0027-05-20087': float(10000)}})
This does not:
evaluator.evaluate({'151': {'clueweb09-en0027-05-20087': 10000}})
There's no valid reason why a matching score cannot be an int, so I think the check at
pytrec_eval/src/pytrec_eval.cpp
Line 270 in 6440d7e
TypeError Traceback (most recent call last)
in ()
----> 1 evaluator.evaluate({'151': {'clueweb09-en0027-05-20087': 10000}})
TypeError: Unable to extract query/object scores.
`
I think that the exact problem is being overwritten by the calling code that checks the return value.
Hi,
Thanks for sharing the tool! It is indeed a very useful one!
I noticed a memory leak problem when running evaluator.evaluate
. It allocates memory slightly higher than the size of the run file, but never releases the memory. To reproduce it, I attached a simple program, with sample qrel
and run
files.
Here is the results of memory profiling. As you can see del res
does not release the memory, leading to allocation of large memory after several runs:
79 55.574 MiB 55.574 MiB @profile
80 def runme():
81
82 57.930 MiB 2.355 MiB qrel = load_reference('qrels.txt')
83 106.215 MiB 48.285 MiB run = load_candidate('run.txt')
84
85
86 107.570 MiB 1.355 MiB evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
87
88 107.570 MiB 0.000 MiB N = 100
89 1320.098 MiB 0.000 MiB for i in range(1,N):
90 1320.098 MiB 18.855 MiB res = evaluator.evaluate(run)
91 1320.098 MiB 0.000 MiB del res
The problem made me go back to old style ac-hoc running of trec_eval. It would be great to get through it, and I am happy to help.
Best,
Navid
providing empty qrel/run file leads to kernel crash!
qrel = {}
run = {}
evaluator = pytrec_eval.RelevanceEvaluator(qrel, pytrec_eval.supported_measures)
results = evaluator.evaluate(run)
It should be catched in anyway.
Dear developers,
I'm using your tool to estimate evaluation metrics for information retrieval research and I got a bit stuck with some behaviour, which doesn't look intuitive for me. Could you please help me to figure out what's going on?
Question:
Suppose I have a qrel file, where the grades are all integers and can take the values: {0, 1, 2, 3}. It's an initial qrel file for RelevanceEvaluator
.
To evaluate the result I take another file where the grades are float and vary from 0 to 100 - [0, 100]. How does the RelevanceEvaluator
behave in that case? Does it use any normalization?
I tried to relabel all {2, 3} values to {1} in the intial qrel file and initialized the RelevanceEvaluator
with such a qrel file with binary grades. And nothing changed... Is it OK?
I used one more run-file for evaluation where grades are float and vary from 0 to 2 - [0, 2] and the results significantly changed with binary initial grades in comparison with {0, 1, 2, 3} initial grades. Why?
Thanks in advance.
Is it right to set qrel relevance scores to sometimg between 1 to 5?
1 means less relevant and 5 means high relevant.
I’m going to use ndcg measure
or additionally, dataframe of the same format.
What if I am predicting a ranked list with the same document ID multiple times in different positions. How can I evaluate nDCG for this using pytrec_eval, given that scores are represented as dictionaries?
when installing with pip or from source code, on the latest MacOS demonstrated the issue:
"warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead".
and later shows the error:
"src/pytrec_eval.cpp:19:10: fatal error: 'algorithm' file not found"
Seems that in the latest MacOS, stdlibc++ is no longer supported and I was expected to use libc++?
When evaluating multiple query relevance (qrel) sets with pytrec_eval, incorrect scores are generated when one or more qrel sets are empty. For example:
qrel = {
'q1': {
'd1': 0,
'd2': 1,
'd3': 0,
},
'q2': {
},
'q3': {
'd2': 1,
'd3': 1,
},
}
evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'})
evaluator.evaluate(some_valid_b)
In this case, the evaluator provides erroneous scores for 'q2' and relations after q2 i.e q3, without raising any warning, which could mislead users.
Heres the error message:
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1131)>
Fetching trec_eval from http://github.com/usnistgov/trec_eval/archive/v9.0.8.tar.gz.
On our proxy we only have a http and https is passed through this, on newer urllib3 this I have read causes a problem.
The fix creates an unsecured connection, without SSL: edit the file: /pytrec_eval/setup.py
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
response = urllib.request.urlopen(REMOTE_TREC_EVAL_URI, context = ctx)
Good Morning,
When I run :"python setup.py install" I get the following error:
"error: command 'gcc' failed with exit status 1. "
I would be grateful if you have a direction what is the problem
The installation log:
install_log.txt
Thanks in advance,
Ortal
Hi!
This outbound call removes the ability of consumers of this package to use it in CI/CD pipelines that prevent eggress.
Would you consider publishing a pre-built wheel that does not require this download?
hello everyone, so basically i'm using windows and i'm trying to install python-terrier and since pytrec_eval is one of the dependencies, there is this error that appears when installing it and break the whole thing:
raise ValueError("path '%s' cannot be absolute" % pathname) ValueError: path '/tmp/tmpgd7izyps/trec_eval-9.0.8/convert_zscores.c' cannot be absolute
any solutions ??
I get the following error when I attempt to install under python 3.7:
src/pytrec_eval.cpp:150:72: error: invalid conversion from ‘const char*’ to ‘char*’ [-fpermissive]
query_document_pairs[pair_idx].docno = PyUnicode_AsUTF8(inner_key);
~~~~~~~~~~~~~~~~^~~~~~~~~~~
error: command 'gcc' failed with exit status 1
Steps to reproduce:
conda create -n py37 python=3.7
source activate py37
pip install pytrec_eval
Works fine under python 3.6.
the same file and the same code, on my laptop gives different results than on google colab (metrics are P_1, P_3, recip_rank, map and ndcg) I did check the precision (running everything on np.float64), yet not the issue.
any suggestion?
The c++ code does not compile on the Mac when installing with pip install pytrec_eval
Output:
src/pytrec_eval.cpp:637:30: error: expected expression
RelevanceEvaluatorType = {
^
11 warnings and 2 errors generated.
error: command '/usr/bin/clang' failed with exit status 1
As a workaround, I made it work by installing gcc/g++ with homebrew and setting the environment so that pip uses gcc instead of clang.
Useful links:
Hi guys,
Any timeline for deploying a new version to PyPi?
trec_eval has nicknames for groups of measures, such as "official" and "all_trec". Could you support this also?
See https://github.com/usnistgov/trec_eval/blob/de6a29f8ba9312c73f978aa9739695aa8ebf48eb/measures.c#L204
Hello author, I have encountered some strange results. The reproduce code is below.
import pytrec_eval
from sklearn.metrics import ndcg_score
import numpy as np
# Create simulated data
qrels_dict = {
'q1': {
'1677128#0': 1,
'doc4': 1,
}
}
results_dict = {
'q1': {
'1677128#0': 2.921875,
'5712855#212': 2.921875,
'4200727#1': 2.890625,
'3750848#3': 2.875,
'doc4': 2.875,
'512030#0': 2.8125,
'512030#1': 2.8125,
'5708398#5': 2.8125,
'6615597#2': 2.796875,
'512030#6': 2.765625,
}
}
# Calculate metrics using pytrec_eval
evaluator = pytrec_eval.RelevanceEvaluator(qrels_dict, {'ndcg_cut.10'})
pytrec_results = evaluator.evaluate(results_dict)
# Prepare data for sklearn
specific_qid = 'q1'
specific_relevances = [
qrels_dict[specific_qid].get(docid, 0)
for docid in results_dict[specific_qid].keys()
]
print(specific_relevances)
specific_scores = [
results_dict[specific_qid][docid] for docid in results_dict[specific_qid].keys()
]
print(specific_scores)
sklearn_specific_ndcg_10 = ndcg_score(
[specific_relevances], [specific_scores], k=10, ignore_ties=False
)
pytrec_specific_ndcg_10 = pytrec_results[specific_qid]['ndcg_cut_10']
# Print results
print(f"qrels_dict['{specific_qid}']:", qrels_dict[specific_qid])
print(f"results_dict['{specific_qid}']:", results_dict[specific_qid])
print(
f"Specific QID: {specific_qid} | pytrec_eval NDCG@10: {pytrec_specific_ndcg_10} | sklearn NDCG@10: {sklearn_specific_ndcg_10}"
)
[1, 0, 0, 0, 1, 0, 0, 0, 0, 0]
[2.921875, 2.921875, 2.890625, 2.875, 2.875, 2.8125, 2.8125, 2.8125, 2.796875, 2.765625]
qrels_dict['q1']: {'1677128#0': 1, 'doc4': 1}
results_dict['q1']: {'1677128#0': 2.921875, '5712855#212': 2.921875, '4200727#1': 2.890625, '3750848#3': 2.875, 'doc4': 2.875, '512030#0': 2.8125, '512030#1': 2.8125, '5708398#5': 2.8125, '6615597#2': 2.796875, '512030#6': 2.765625}
Specific QID: q1 | pytrec_eval NDCG@10: 0.6509209298071326 | sklearn NDCG@10: 0.7506329176709435
import pytrec_eval
from sklearn.metrics import ndcg_score
import numpy as np
# Create simulated data
qrels_dict = {
'q1': {
'1677128#0': 1,
'1677128#1': 1,
}
}
results_dict = {
'q1': {
'1677128#0': 2.921875,
'5712855#212': 2.921875,
'4200727#1': 2.890625,
'3750848#3': 2.875,
'1677128#1': 2.875, # i only changed key name then i get a different ndcg
'512030#0': 2.8125,
'512030#1': 2.8125,
'5708398#5': 2.8125,
'6615597#2': 2.796875,
'512030#6': 2.765625,
}
}
# Calculate metrics using pytrec_eval
evaluator = pytrec_eval.RelevanceEvaluator(qrels_dict, {'ndcg_cut.10'})
pytrec_results = evaluator.evaluate(results_dict)
# Prepare data for sklearn
specific_qid = 'q1'
specific_relevances = [
qrels_dict[specific_qid].get(docid, 0)
for docid in results_dict[specific_qid].keys()
]
print(specific_relevances)
specific_scores = [
results_dict[specific_qid][docid] for docid in results_dict[specific_qid].keys()
]
print(specific_scores)
sklearn_specific_ndcg_10 = ndcg_score(
[specific_relevances], [specific_scores], k=10, ignore_ties=False
)
pytrec_specific_ndcg_10 = pytrec_results[specific_qid]['ndcg_cut_10']
# Print results
print(f"qrels_dict['{specific_qid}']:", qrels_dict[specific_qid])
print(f"results_dict['{specific_qid}']:", results_dict[specific_qid])
print(
f"Specific QID: {specific_qid} | pytrec_eval NDCG@10: {pytrec_specific_ndcg_10} | sklearn NDCG@10: {sklearn_specific_ndcg_10}"
)
[1, 0, 0, 0, 1, 0, 0, 0, 0, 0]
[2.921875, 2.921875, 2.890625, 2.875, 2.875, 2.8125, 2.8125, 2.8125, 2.796875, 2.765625]
qrels_dict['q1']: {'1677128#0': 1, '1677128#1': 1}
results_dict['q1']: {'1677128#0': 2.921875, '5712855#212': 2.921875, '4200727#1': 2.890625, '3750848#3': 2.875, '1677128#1': 2.875, '512030#0': 2.8125, '512030#1': 2.8125, '5708398#5': 2.8125, '6615597#2': 2.796875, '512030#6': 2.765625}
Specific QID: q1 | pytrec_eval NDCG@10: 0.6240505200038379 | sklearn NDCG@10: 0.7506329176709435
import pytrec_eval
from sklearn.metrics import ndcg_score
import numpy as np
# Create simulated data
qrels_dict = {
'q1': {
'1677128#0': 1,
'doc4': 1,
}
}
results_dict = {
'q1': {
'1677128#0': 2.921875,
'5712855#212': 2.921875,
'4200727#1': 2.890625,
'3750848#3': 2.875,
'doc4': 2.875,
'512030#0': 2.8125,
'512030#1': 2.8125,
'5708398#5': 2.8125,
'6615597#2': 2.796875,
'512030#6': 2.765625,
}
}
# Calculate metrics using pytrec_eval
evaluator = pytrec_eval.RelevanceEvaluator(qrels_dict, {'ndcg_cut.10'})
pytrec_results = evaluator.evaluate(results_dict)
# Prepare data for sklearn
specific_qid = 'q1'
specific_relevances = [
qrels_dict[specific_qid].get(docid, 0)
for docid in results_dict[specific_qid].keys()
]
print(specific_relevances)
specific_scores = [
results_dict[specific_qid][docid] for docid in results_dict[specific_qid].keys()
]
print(specific_scores)
sklearn_specific_ndcg_10 = ndcg_score(
[specific_relevances], [specific_scores], k=10, ignore_ties=True
)
pytrec_specific_ndcg_10 = pytrec_results[specific_qid]['ndcg_cut_10']
# Print results
print(f"qrels_dict['{specific_qid}']:", qrels_dict[specific_qid])
print(f"results_dict['{specific_qid}']:", results_dict[specific_qid])
print(
f"Specific QID: {specific_qid} | pytrec_eval NDCG@10: {pytrec_specific_ndcg_10} | sklearn NDCG@10: {sklearn_specific_ndcg_10}"
)
[1, 0, 0, 0, 1, 0, 0, 0, 0, 0]
[2.921875, 2.921875, 2.890625, 2.875, 2.875, 2.8125, 2.8125, 2.8125, 2.796875, 2.765625]
qrels_dict['q1']: {'1677128#0': 1, 'doc4': 1}
results_dict['q1']: {'1677128#0': 2.921875, '5712855#212': 2.921875, '4200727#1': 2.890625, '3750848#3': 2.875, 'doc4': 2.875, '512030#0': 2.8125, '512030#1': 2.8125, '5708398#5': 2.8125, '6615597#2': 2.796875, '512030#6': 2.765625}
Specific QID: q1 | pytrec_eval NDCG@10: 0.6509209298071326 | sklearn NDCG@10: 0.6509209298071325
ERROR: Could not build wheels for pytrec-eval, which is required to install pyproject.toml-based projects
When I try to install pytrec_eval via pip install pytrec_eval, I am getting this error.
ERROR: Could not build wheels for pytrec_eval, which is required to install pyproject.toml-based projects
win 10 and python 3.9.0
I am wondering the value range for document score for run file should be within [0, 1], or it can be any value including negative floating point as well ?
The -c
option in trec_eval
does the following:
--complete_rel_info_wanted:
-c: Average over the complete set of queries in the relevance judgements
instead of the queries in the intersection of relevance judgements
and results. Missing queries will contribute a value of 0 to all
evaluation measures (which may or may not be reasonable for a
particular evaluation measure, but is reasonable for standard TREC
measures.) Default is off.
Although the default in trec_eval
is off, I think it would be prudent to default this value to on (and maybe give the user an option to turn it off). Without this, a user may accidentally average over an incomplete set of queries, e.g., if their engine doesn't return any results for a given query.
It doesn't look like this is as simple as setting:
self->epi_.average_complete_flag = 1;
because the setting only affects trec_eval
's averages, not the individual per-query scores. A fix could be modifying the run dict before sending it to the relevance assessor, adding in any missing queries pointing to empty dicts.
I am getting this error when running some evalution task with MTEB.
It seems like it has something to do with pytrec. The error occurs in the shared library (pytrec_eval_ext.cpython-310-x86_64-linux-gnu.so).
Interestingly I copied over the shared library from a different machine with similar configuratiton and the same version of pytrec_eval installed and it seems to solve the issue. Nevertheless, it would be good to find out what is going wrong.
Here is the trace:
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/mguenther/finetuner-large-scale-training/run_mteb.py", line 42, in <module>
evaluation.run(
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/mteb/evaluation/MTEB.py", line 398, in run
raise e
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/mteb/evaluation/MTEB.py", line 364, in run
results, tick, tock = self._run_eval(
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/mteb/evaluation/MTEB.py", line 260, in _run_eval
results = task.evaluate(model, split, output_folder=output_folder, **kwargs)
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 273, in evaluate
scores[hf_subset] = self._evaluate_subset(
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 311, in _evaluate_subset
ndcg, _map, recall, precision = retriever.evaluate(
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 538, in evaluate
evaluator = pytrec_eval.RelevanceEvaluator(
File "/home/mguenther/finetuner-large-scale-training/venv/lib/python3.10/site-packages/pytrec_eval/__init__.py", line 59, in __init__
super().__init__(query_relevance=query_relevance, measures=measures, relevance_level=relevance_level, judged_docs_only_flag=judged_docs_only_flag)
TypeError: Expected object_relevance_per_qid dictionary and measures set.
The RelevanceEvaluator is initialized in the following way:
evaluator = pytrec_eval.RelevanceEvaluator(
qrels, {map_string, ndcg_string, recall_string, precision_string}
)
with qrles
being:
defaultdict(<class 'dict'>, {'1': {'31715818': 1}, '3': {'14717500': 1}, '5': {'13734012': 1}, '13': {'1606628': 1}, '36': {'5152028': 1, '11705328': 1}, '42': {'18174210': 1}, '48': {'13734012': 1}, '49': {'5953485': 1}, '50': {'12580014': 1}, '51': {'45638119': 1}, '53': {'45638119': 1}, '54': {'49556906': 1}, '56': {'4709641': 1}, '57': {'4709641': 1}, '70': {'5956380': 1, '4414547': 1}, '72': {'6076903': 1}, '75': {'4387784': 1}, '94': {'1215116': 1}, '99': {'18810195': 1}, '100': {'4381486': 1}, '113': {'6157837': 1}, '115': {'33872649': 1}, '118': {'6372244': 1}, '124': {'4883040': 1}, '127': {'21598000': 1}, '128': {'8290953': 1}, '129': {'27768226': 1}, '130': {'27768226': 1}, '132': {'7975937': 1}, '133': {'38485364': 1, '6969753': 1, '17934082': 1, '16280642': 1, '12640810': 1}, '137': {'26016929': 1}, '141': {'6955746': 1, '14437255': 1}, '142': {'10582939': 1}, '143': {'10582939': 1}, '146': {'10582939': 1}, '148': {'1084345': 1}, '163': {'18872233': 1}, '171': {'12670680': 1}, '179': {'16322674': 1, '27123743': 1, '23557241': 1, '17450673': 1}, '180': {'16966326': 1}, '183': {'12827098': 1}, '185': {'18340282': 1}, '198': {'2177022': 1}, '208': {'13519661': 1}, '212': {'22038539': 1}, '213': {'13625993': 1}, '216': {'21366394': 1}, '217': {'21366394': 1}, '218': {'21366394': 1}, '219': {'21366394': 1}, '230': {'3067015': 1}, '232': {'10536636': 1}, '233': {'4388470': 1}, '236': {'4388470': 1}, '237': {'4942718': 1}, '238': {'2251426': 1}, '239': {'14079881': 1}, '248': {'1568684': 1}, '249': {'1568684': 1}, '261': {'1122279': 1, '10697096': 1}, '268': {'970012': 1}, '269': {'970012': 1}, '274': {'11614737': 1}, '275': {'4961038': 1, '14241418': 1, '14819804': 1}, '279': {'14376683': 1}, '294': {'10874408': 1}, '295': {'20310709': 1}, '298': {'39381118': 1}, '300': {'3553087': 1}, '303': {'4388470': 1}, '312': {'6173523': 1}, '314': {'4347374': 1}, '324': {'2014909': 1}, '327': {'17997584': 1}, '338': {'23349986': 1}, '343': {'7873737': 1, '5884524': 1}, '350': {'16927286': 1}, '354': {'8774475': 1}, '362': {'38587347': 1}, '380': {'19005293': 1}, '384': {'13770184': 1}, '385': {'9955779': 1, '9767444': 1}, '386': {'16495649': 1}, '388': {'1148122': 1}, '399': {'791050': 1}, '410': {'14924526': 1}, '411': {'14924526': 1}, '415': {'6309659': 1}, '421': {'11172205': 1}, '431': {'28937856': 1}, '436': {'14637235': 1}, '437': {'18399038': 1}, '439': {'4423559': 1}, '440': {'4423559': 1}, '443': {'10165258': 1}, '452': {'12804937': 1, '464511': 1}, '475': {'18678095': 1}, '478': {'14767844': 1}, '491': {'56893404': 1}, '501': {'17930286': 1}, '502': {'13071728': 1}, '507': {'30774694': 1}, '508': {'13980338': 1}, '513': {'13230773': 1}, '514': {'16256507': 1}, '516': {'29564505': 1}, '517': {'15663829': 1}, '521': {'34873974': 1}, '525': {'13639330': 1}, '527': {'3863543': 1}, '528': {'5476778': 1}, '532': {'12991445': 1}, '533': {'12991445': 1}, '535': {'39368721': 1}, '536': {'16056514': 1}, '539': {'13282296': 1}, '540': {'11886686': 1, '25007443': 1}, '544': {'24221369': 1}, '549': {'9433958': 1}, '551': {'33499189': 1}, '552': {'1471041': 1}, '554': {'1049501': 1}, '560': {'40096222': 1}, '569': {'23460562': 1}, '575': {'10300888': 1}, '577': {'5289038': 1}, '578': {'8764879': 1}, '587': {'16999023': 1}, '589': {'10984005': 1}, '593': {'19675911': 1}, '597': {'12779444': 1, '36355784': 1, '25742130': 1}, '598': {'25742130': 1}, '613': {'9638032': 1}, '619': {'20888849': 1, '2565138': 1}, '623': {'17000834': 1}, '628': {'24512064': 1}, '636': {'24294572': 1}, '637': {'25649714': 1}, '641': {'5912283': 1, '31554917': 1}, '644': {'13619127': 1}, '649': {'12789595': 1}, '659': {'1215116': 1}, '660': {'1215116': 1}, '674': {'2095573': 1}, '684': {'4942718': 1}, '690': {'18750453': 1}, '691': {'10991183': 1}, '692': {'24088502': 1}, '693': {'24088502': 1}, '700': {'4350400': 1}, '702': {'4350400': 1}, '715': {'18421962': 1}, '716': {'18421962': 1}, '718': {'17587795': 1}, '721': {'1834762': 1}, '723': {'5531479': 1}, '727': {'7521113': 1}, '728': {'7521113': 1, '36444198': 1}, '729': {'26851674': 1}, '742': {'32159283': 1}, '743': {'32159283': 1}, '744': {'8460275': 1}, '756': {'2831620': 1}, '759': {'1805641': 1}, '768': {'6421792': 1}, '770': {'15476777': 1}, '775': {'32275758': 1}, '781': {'24338780': 1}, '783': {'40632104': 1}, '784': {'2356950': 1}, '785': {'12471115': 1}, '793': {'8551160': 1}, '800': {'22543403': 1}, '805': {'22180793': 1}, '808': {'36606083': 1}, '811': {'19799455': 1}, '814': {'33387953': 1}, '820': {'8646760': 1}, '821': {'8646760': 1}, '823': {'15319019': 1}, '830': {'1897324': 1}, '831': {'1897324': 1}, '832': {'30303335': 1}, '834': {'5483793': 1}, '837': {'15928989': 1}, '839': {'1469751': 1}, '845': {'17741440': 1}, '847': {'16787954': 1}, '852': {'13843341': 1}, '859': {'1982286': 1}, '870': {'195689316': 1}, '873': {'1180972': 1, '19307912': 1, '27393799': 1, '29025270': 1, '3315558': 1}, '879': {'8426046': 1}, '880': {'8426046': 1}, '882': {'14803797': 1}, '887': {'18855191': 1}, '903': {'10648422': 1}, '904': {'7370282': 1}, '907': {'6923961': 1}, '911': {'11254556': 1}, '913': {'3203590': 1}, '914': {'3203590': 1}, '921': {'1642727': 1}, '922': {'17077004': 1}, '936': {'5483793': 1}, '956': {'12956194': 1}, '957': {'123859': 1}, '960': {'8780599': 1}, '967': {'2119889': 1, '8997410': 1}, '971': {'46695481': 1, '27873158': 1, '28617573': 1, '9764256': 1}, '975': {'5304891': 1}, '982': {'2988714': 1}, '985': {'6828370': 1}, '993': {'16472469': 1}, '1012': {'9745001': 1}, '1014': {'6277638': 1}, '1019': {'11603066': 1}, '1020': {'9433958': 1}, '1021': {'9433958': 1}, '1024': {'5373138': 1}, '1029': {'13923140': 1, '13940200': 1, '11899391': 1}, '1041': {'25254425': 1, '16626264': 1}, '1049': {'12486491': 1}, '1062': {'20381484': 1}, '1086': {'39281140': 1}, '1088': {'37549932': 1}, '1089': {'17628888': 1}, '1099': {'7662206': 1}, '1100': {'7662206': 1}, '1104': {'3898784': 1}, '1107': {'20532591': 1}, '1110': {'13770184': 1}, '1121': {'4456756': 1}, '1130': {'17997584': 1}, '1132': {'33499189': 1, '9283422': 1}, '1137': {'33370': 1}, '1140': {'12009265': 1}, '1144': {'10071552': 1}, '1146': {'13906581': 1}, '1150': {'11369420': 1}, '1163': {'15305881': 1}, '1175': {'31272411': 1}, '1179': {'31272411': 1}, '1180': {'31272411': 1}, '1185': {'16737210': 1}, '1187': {'52873726': 1}, '1191': {'30655442': 1}, '1194': {'11419230': 1}, '1196': {'25649714': 1}, '1197': {'25649714': 1}, '1199': {'16760369': 1}, '1200': {'3441524': 1}, '1202': {'3475317': 1}, '1204': {'31141365': 1}, '1207': {'18909530': 1}, '1213': {'14407673': 1}, '1216': {'24142891': 1}, '1221': {'19736671': 1}, '1225': {'9650982': 1}, '1226': {'13777138': 1}, '1232': {'13905670': 1}, '1241': {'4427392': 1}, '1245': {'7662395': 1}, '1259': {'24341590': 1}, '1262': {'44172171': 1}, '1266': {'37480103': 1}, '1270': {'13900610': 1}, '1271': {'13768432': 1}, '1272': {'17081238': 1}, '1273': {'11041152': 1}, '1274': {'12428814': 1, '27731651': 1, '4406819': 1}, '1278': {'11335781': 1}, '1279': {'11335781': 1}, '1280': {'4387784': 1}, '1281': {'4387784': 1}, '1282': {'23649163': 1}, '1290': {'4687948': 1}, '1292': {'56893404': 1}, '1298': {'11718220': 1}, '1303': {'12631697': 1}, '1316': {'27910499': 1}, '1319': {'16284655': 1}, '1320': {'16284655': 1}, '1332': {'5304891': 1}, '1335': {'27910499': 1}, '1336': {'27910499': 1}, '1337': {'20231138': 1}, '1339': {'15482274': 1}, '1344': {'9559146': 1}, '1352': {'12885341': 1}, '1359': {'11614737': 1}, '1362': {'8290953': 1}, '1363': {'8290953': 1}, '1368': {'2425364': 1}, '1370': {'2425364': 1}, '1379': {'16322674': 1, '27123743': 1, '23557241': 1, '17450673': 1}, '1382': {'17755060': 1}, '1385': {'306006': 1}, '1389': {'23895668': 1}, '1395': {'17717391': 1}})
and {map_string, ndcg_string, recall_string, precision_string}
:
{'ndcg_cut.1,3,5,10,20,100,1000', 'recall.1,3,5,10,20,100,1000', 'map_cut.1,3,5,10,20,100,1000', 'P.1,3,5,10,20,100,1000'}
Both machines have pytrec_eval verison 0.5, setuptools 59.6.0, Python 3.10.12, Ubuntu 22.04, and gcc 11.4.0
Would it be possible to make a binary build for Windows?
More information at:
https://packaging.python.org/guides/packaging-binary-extensions/#binary-extensions-for-windows
It might be possible to use a Github action for building the release.
Hello,
I tried installing pytrec_eval under m1 and the installation work, but when I try to import it I get this issues.
Traceback (most recent call last):
File "/Users/tcarvalho/Library/Caches/pypoetry/virtualenvs/evaluation-suite-iSLcvJiA-py3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-b628730d1a82>", line 1, in <module>
import pytrec_eval
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/Users/tcarvalho/Library/Caches/pypoetry/virtualenvs/evaluation-suite-iSLcvJiA-py3.10/lib/python3.10/site-packages/pytrec_eval/__init__.py", line 7, in <module>
from pytrec_eval_ext import RelevanceEvaluator as _RelevanceEvaluator
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
ImportError: dlopen(/Users/tcarvalho/Library/Caches/pypoetry/virtualenvs/evaluation-suite-iSLcvJiA-py3.10/lib/python3.10/site-packages/pytrec_eval_ext.cpython-310-darwin.so, 0x0002): tried: '/Users/tcarvalho/Library/Caches/pypoetry/virtualenvs/evaluation-suite-iSLcvJiA-py3.10/lib/python3.10/site-packages/pytrec_eval_ext.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/tcarvalho/Library/Caches/pypoetry/virtualenvs/evaluation-suite-iSLcvJiA-py3.10/lib/python3.10/site-packages/pytrec_eval_ext.cpython-310-darwin.so' (no such file), '/Users/tcarvalho/Library/Caches/pypoetry/virtualenvs/evaluation-suite-iSLcvJiA-py3.10/lib/python3.10/site-packages/pytrec_eval_ext.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
My understanding is that the underlying C packages aren't supported for M1. I tried installing directly from the tar file, but I have the same issue.
Hi,
Do you guys have any tips to have a faster metric computation ? I'm using the MAP and it's really slow as I need to do it quite often.
Kudos for the great interface!
I have been running some evaluations where I obtain the NDCG@10 and NDCG@100 metrics. I noticed that if I run the evaluator twice, the second time only one of the metrics appears. Here is a simple example, modified from simple.py:
import pytrec_eval
import json
qrel = {
'q1': {
'd1': 0,
'd2': 1,
'd3': 0,
},
'q2': {
'd2': 1,
'd3': 1,
},
}
run = {
'q1': {
'd1': 1.0,
'd2': 0.0,
'd3': 1.5,
},
'q2': {
'd1': 1.5,
'd2': 0.2,
'd3': 0.5,
}
}
evaluator = pytrec_eval.RelevanceEvaluator(
qrel, {'ndcg_cut_10', 'ndcg_cut_100'})
print(json.dumps(evaluator.evaluate(run), indent=1))
# Just calling .evaluate() again, but run could be different
print(json.dumps(evaluator.evaluate(run), indent=1))
Output:
{
"q1": {
"ndcg_cut_10": 0.5,
"ndcg_cut_100": 0.5
},
"q2": {
"ndcg_cut_10": 0.6934264036172708,
"ndcg_cut_100": 0.6934264036172708
}
}
{
"q1": {
"ndcg_cut_10": 0.5
},
"q2": {
"ndcg_cut_10": 0.6934264036172708
}
}
I am running an experinment in PyTerrier and i am trying to measure the iprec_at_recall at all recall values. However, i get the following error when i run the experinment ( works fine with other measures such as ndcg etc ).
trec_eval: duplicate cutoffs detected
python3: src/pytrec_eval.cpp:634: PyObject* RelevanceEvaluator_evaluate(RelevanceEvaluator*, PyObject*): Assertion 'te_trec_measures[measure_idx]->eval_index >=0' failed.
Where can I find the list of metrics and their descriptions that pytrec_eval supports. I tried looking at the trec_eval repo as well and it doesn't seem to list the metric names and associated descriptions in a clear place. For example I tried passing "precision" and got an unsupported measure error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.