meeliskull / prg

Software to create Precision-Recall-Gain curves and calculate area under the curve

prg: software to create Precision-Recall-Gain curves and calculate area under the curve

What are the Precision-Recall-Gain curves?

Please see http://www.cs.bris.ac.uk/~flach/PRGcurves/.

Software

This software is available in the following forms:

• R-package
• Python-package
• Matlab-package
• Package for Java is under development
• We are happy to implement more forms, please e-mail us at [email protected] to suggest your favourite form to be added.

Example

An example in Python is available here: Jupyter notebook

Authors

This software has been written by Meelis Kull, Telmo de Menezes e Silva Filho and Miquel Perello Nieto, based on work by Peter Flach and Meelis Kull, see http://www.cs.bris.ac.uk/~flach/PRGcurves/.

prg's Issues

Very large PRG AUC values even for poor classifiers

Hi, when dealing with a binary classification problem, I find the AUPRG returned from prg.calc_auprg can be very close to 1 even though the classifier can't classify the positive class well. This result doesn't seem to make sense as AUPRG~1 should occur when the classification is perfectly right. Is there additional calibration in order to use AUPRG to rank models? Or is it a bug?

Example code to reproduce the results:

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import auc, average_precision_score, PrecisionRecallDisplay
from prg import prg

X, y = make_classification(n_samples=2000, n_classes=2, weights=[0.99, 0.01], random_state=1)
trainX, testX, trainy, testy = train_test_split(X, y, test_size=0.5, random_state=2, stratify=y)
print(f'Dataset: Class0={len(y[y==0])}, Class1={len(y[y==1])}')
print(f'Train: Class0={len(trainy[trainy==0])}, Class1={len(trainy[trainy==1])}')
print(f'Test:  Class0={len(testy[testy==0])}, Class1={len(testy[testy==1])}') 

model = LogisticRegression(solver='lbfgs')
model.fit(trainX, trainy)
y_pred = model.predict_proba(testX)[:, 1]

# calculate the PRG auc
prg_curve = prg.create_prg_curve(testy, y_pred)
auprg = prg.calc_auprg(prg_curve)
prg.plot_prg(prg_curve);

# calculate the precision-recall auc
precision, recall, _ = precision_recall_curve(testy, y_pred)
prauc = auc(recall, precision)
ap = average_precision_score(testy, y_pred)
PrecisionRecallDisplay(precision=precision, recall=recall, average_precision=ap, estimator_name='logreg').plot()
print(f'AUPRG:{auprg}, AUPR:{prauc}, Average Precision:{ap}')

The returned results are

AUPRG:0.9892310395473074, AUPR:0.2668421579811339, Average Precision:0.27709872964106963

PRG-curve:

PR-curve:

I am using Python 3.7.7. The version of pyprg and sklearn in my environment are respectively 0.1.1b7 and 0.22.2.

Python: inconsistent returned types for calc_auprg (int, float)

The returned types for function calc_auprg are not consitent, it may be convenient to force and return always floats.

from prg import prg
import numpy as np

y_true = np.array([0, 0, 1, 1], dtype='int')
scores = np.arange(4, 1, -1)
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)
print(auprg)
print(type(auprg))


y_true = np.array([0, 1, 0], dtype='int')
scores = np.arange(3, 1, -1)
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)
print(auprg)
print(type(auprg))

Output

0
<class 'int'>
0.0
<class 'numpy.float64'>

Python: ValueError exception when only one positive label has the lowest score

The following example contains two cases that raise the exception shown below. I am not sure if it happens only when there is only one positive sample and it has the lowest output score. Need to investigate further.

from prg import prg
import numpy as np

y_true = np.array([1, 1, 0, 0], dtype='int')
scores = np.arange(4, 1, -1)
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)

y_true = np.array([0, 0, 1, 1], dtype='int')
scores = np.arange(4, 1, -1)
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)

y_true = np.array([0, 0, 1, 0], dtype='int')
scores = np.arange(4, 1, -1)
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)

y_true = np.array([0, 0, 0, 1], dtype='int')
scores = np.arange(4, 1, -1)
# This one raises an exception
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)

y_true = np.array([0, 1, 0], dtype='int')
scores = np.arange(3, 1, -1)
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)

y_true = np.array([0, 0, 1], dtype='int')
scores = np.arange(3, 1, -1)
# This one raises an exception
prg_curve = prg.create_prg_curve(y_true, scores)
auprg = prg.calc_auprg(prg_curve)
print(auprg)
type(auprg)

Raised exception

Traceback (most recent call last):
  File "test_exception.py", line 33, in <module>
    prg_curve = prg.create_prg_curve(y_true, scores)
  File "/home/miquel/git/uob/conditional_sampler/venv/lib/python3.6/site-packages/prg/prg.py", line 212, in create_prg_curve
    points = _create_crossing_points(points, n_pos, n_neg)
  File "/home/miquel/git/uob/conditional_sampler/venv/lib/python3.6/site-packages/prg/prg.py", line 126, in _create_crossing_points
    j = np.amin(np.where(points['recall_gain'] >= 0)[0])
  File "<__array_function__ internals>", line 6, in amin
  File "/home/miquel/git/uob/conditional_sampler/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 2831, in amin
    keepdims=keepdims, initial=initial, where=where)
  File "/home/miquel/git/uob/conditional_sampler/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 87, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

Cannot install the package anymore

I get the following error while trying to install the package:

# install the prg package
devtools::install_github("meeliskull/prg/R_package/prg")

Error in install_github("meeliskull/prg/R_package/prg") : 
  lazy-load database 'C:/Users/anwar/Documents/R/win-library/4.1/pkgbuild/R/pkgbuild.rdb' is corrupt
In addition: Warning messages:
1: In install_github("meeliskull/prg/R_package/prg") :
  restarting interrupted promise evaluation
2: In install_github("meeliskull/prg/R_package/prg") :
  internal error -3 in R_decompress1