qbeer / coco-froc-analysis

FROC analysis for COCO detections for Detectron(2) and OpenMMLab

About me

I was studying Physics for my BSc and MSc degrees and currently I am a Computer Science PhD student in applied deep learning for medical imaging. I am also married and I run a technical blog. I have various interests in the field of deep learning and especially in computer vision but I enjoy reading about advancements in other subfields as well.

🔭 Recently we got 2nd place and took the first prize of the Covid CXR Hackathon - Artificial Intelligence for Covid-19 prognosis: aiming at accuracy and explainability. In this challenge we built the most accurate model that took the prize and also submitted a sophisticated Bahdanau-attention based model to create explainability attention maps.

🔭 I’m currently working on computer aided diagnostics for pathology regarding colorectal cancer screening. We analyze and create new methods for automated whole-slide segmentation. I also particated, in a team of two, in a DreamChallenge regarding automated scoring of joint damage and got 2nd (2nd subchallenge) and 3rd (1st and 3rd subchallenges) places.

📫 How to reach me: you can reach my on my personal email that is present on my webpage or through here

⚡ Social: Twitter LinkedIn Facebook

@Regards, Alex

coco-froc-analysis's Issues

[bug_report] no gt image will lead to no "FP" and "cost_matrix" error

Thanks for your work and share!

If there is picture without any gt bbox, there will be error no "FP" or "cost_matrix" .
so I add FP in cat define:

cat["id"]: {
            "name": cat["name"],
            "LL": 0,
            "NL": 0,
            "FP":0,
            "n_images": [],
            "n_lesions": 0,
        }

and add null cost_matrix if n_gt == 0:

if n_gt == 0:
    if n_pr == 0:
        continue
    stats[cat['id']]['FP'] += n_pr
    cost_matrix = [[]]

No error shows anymore.

use_iou=True not working?

Thank you for your work and share!
I have a test json from coco dataset.
when I set use_iou=True, and play with iou_thres, I get the same FROC picture everytime.
It looks like use_iou=True is not working.

here is the gt and predict json file I am using.
https://drive.google.com/file/d/1Z8E3kmQ_BYIoZoPLX4xbFOp3Rd2h7wb4/view?usp=sharing
https://drive.google.com/file/d/1eoqKkpKSYxUd4NCWy-WkEbCKYbaQdIGa/view?usp=sharing

and the code I am running this is:

from froc_analysis import generate_froc_curve
generate_froc_curve(gt_ann='/data/testcoco100_c0_gt.json',
                    pr_ann='/data/testcoco100_c0_pd.json',
                    use_iou=True, iou_thres=0.8, n_sample_points=75,
                    plot_title='FROC', plot_output_path='dummy08.png')

generate_froc_curve(gt_ann='/data/testcoco100_c0_gt.json',
                    pr_ann='/data/testcoco100_c0_pd.json',
                    use_iou=True, iou_thres=0.000001, n_sample_points=75,
                    plot_title='FROC', plot_output_path='dummy0000001.png')

Could you take a look on this issue?
Thank you so much!

Updates are required in the usage

Your repository is handy, thank you for your contribution!
I find the usage in the README.md needs to be updated
I install your package by python setup.py install. When using the following code shown in the usage,
from froc_analysis import generate_froc_curve, generate_bootstrap_curves
It turns out to be ImportError cannot import name 'generate_bootstrap_curves'. I think this is because the code has been upgraded and the usage has not been updated. I found the new function generate_bootstrap_froc_curves can be imported and it seems to have the same function as the 'generate_bootstrap_curves'.
I am interested in the new function count_curve, is this used to plot the Precision-Recall Curve (PR curve)? I would be grateful if you could give me some information about it. I am looking forward to your reply.

Which documtentation or paper refered?

Thank you for your contribution.

I wonder which documentation(or paper) was referred for this code.

(to understand how to calculate FROC metric.)

Error with file-loading

The current file loading should be changed to this to not produce a decode error:

def load_json_from_file(file_path):
    with Path(file_path).open('r', encoding='utf-8') as fp:
        data = json.load(fp)
    return data

issue with the formula

for the use_IOU function, I dont understand why you need to get the uniform distribution as in the formula, and I also believe that you use wrong ()

also I think when you use linear sum assignment function, row_ind = len(gt) which is not our desired