Maybe wrong in F_measure calculation on openset about openlongtailrecognition-oltr

[zhmiao]

Hello @tuobay , thanks for asking. The F-measurement we use is according to this paper: https://arxiv.org/pdf/1511.06233.pdf , where false-positive is defined as "incorrect classifications on the validation set". I think the validation set in this paper are from seen classes, so that for false positives, labels should not equal to -1. Does that make sense?

[saurabhsharma1993]

@zhmiao a related question to above: I tried changing the open set threshold as in Fig 8b) of the paper, but I'm getting different results. Specifically, I get a monotonically increasing sequence of F-measure values, rather than decreasing, on increasing open set threshold from 0 to 1. Any ideas why ?

[saurabhsharma1993]

This is what I get for ImageNet-LT, using the Stage-2 model (its not the latest) :

F_measure (with threshold 0.00) :0.3993
F_measure (with threshold 0.10) :0.4532
F_measure (with threshold 0.20) :0.5793
F_measure (with threshold 0.30) :0.6842
F_measure (with threshold 0.40) :0.7646
F_measure (with threshold 0.50) :0.8261
F_measure (with threshold 0.60) :0.8696
F_measure (with threshold 0.70) :0.9095
F_measure (with threshold 0.80) :0.9348
F_measure (with threshold 0.90) :0.9475

[tuobay]

I think the TP + FP + TN + FN = num of all test samples

So the class (label = -1 and pred = positive) should be put into FP.

[saurabhsharma1993]

@zhmiao @liuziwei7 awaiting your response

[zhmiao]

Hello @tuobay @ssfootball04 . Thank you very much for the discussion. We think the problem is indeed false positive calculation. The new false positive is : false_pos += 1 if preds[i] != labels[i] and labels[i] != -1 else 0 . We removed the preds[i] != -1 since it actually does not make sense. According to the paper we cited, " false positives are incorrect classifications on the validation set". So we think the current calculation is correct. After removing this, the F-measure numbers are normal. We think the reported F-measure numbers are a little bit higher than actual numbers for all baselines. We will update it as soon as possible. I have pushed the new code already. Please check it out. Thanks again.

[zhmiao]

Since it has been a while, I will close this issue for now. Please feel free to re-open this issue if any questions raised. Thanks.