asharakeh / pod_compare Goto Github PK

Hi,

The corresponding paper to this code notes that PDQ "is evaluated at a single classification score threshold requiring practitioners to filter low scoring output detection results prior to evaluation", and the official COCO PDQ evaluation code has an argument "label_threshold" for this purpose. In the PDQ paper, they loosely use 0.5 as this threshold. I'm trying to calculate PDQ on models trained from this code and I can't find anything about what y'all set this threshold to, so I'm wondering if you used 0.5 as well or something else to produce your paper results.

Thanks!

When I run the training script, I met the following error "AttributeError: 'ProbabilisticRetinaNet' object has no attribute 'loss_normalizer_momentum'". How do you fix this error, thank you!

Hi @asharakeh , thanks for releasing your code! Congratulations on the great work.

I have a problem with the calculation of uncertainty error in your code.

For example, when computing regression minimum uncertainty error, you use
reg_u_errors = 0.5 * ((sorted_gt_idxs_tp.sum(0) - tp_cum_sum) / sorted_gt_idxs_tp.sum(0)) + 0.5 * (fp_cum_sum / sorted_gt_idxs_fp.sum(0))
the first numerator is not the number of TPs whose uncertainties are larger than an uncertainty threshold, and the second numerator is not the number of FPs whose uncertainties are smaller than the uncertainty threshold, besides, I did not find the uncertainty thresholds you set to get the uncertainty error.

Am I missing something, or you defined the uncertainty error differently here?

Hey, first of all, thank you for your excellent work.
I have a question regarding the matching of your evaluation which I might have overlooked in the paper or code.
A True Positive (TP) is defined as usual by a certain threshold and I am really curious about how you handle duplicate predictions:

1. Do you match multiple predictions to a single ground truth (gt) object so you have multiple TPs per gt? Or is any duplicate considered as a False Positive (FP)?
2. Similar to the first questions: Can a single prediction have multiple ground truths?
3. Are you only using TPs to evaluate calibration errors and scoring rules? If so, do you then assess the accuracy (of matched boxes) to determine the marginal calibration error?

I met the following error "AttributeError: 'ProbabilisticRetinaNet' object has no attribute 'self.inference'". How do you fix this error, thank you!

Hi @asharakeh, really interesting work.
I want to know if one can do inference on their own custom data?

Running the convert script from the project directory
python /home/richard.tanai/codes/pod_compare/src/core/datasets/convert_bdd_to_coco.py --dataset-dir ~/datasets/bdd/

Gives the error
Traceback (most recent call last):

File "/home/richard.tanai/codes/pod_compare/src/core/datasets/convert_bdd_to_coco.py", line 159, in
main(args)
File "/home/richard.tanai/codes/pod_compare/src/core/datasets/convert_bdd_to_coco.py", line 110, in main
training_image_list, training_annotation_list = create_coco_lists(
File "/home/richard.tanai/codes/pod_compare/src/core/datasets/convert_bdd_to_coco.py", line 53, in create_coco_lists
if annotation['category'] in category_keys:
KeyError: 'category'

I have tried to check the dict by printing it before the error occurs around line 51

        annotations = grouped_per_frame[frame]
        for annotation in annotations:
            print(annotation)
            if annotation['category'] in category_keys:

The result of the print is as follows:
{'name': '0000f77c-6257be58.jpg', 'attributes': {'weather': 'clear', 'scene': 'city street', 'timeofday': 'daytime'}, 'timestamp': 10000, 'labels': [{'category': 'traffic light', 'attributes': {'occluded': False, 'truncated': False, 'trafficLightColor': 'green'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 1125.902264, 'y1': 133.184488, 'x2': 1156.978645, 'y2': 210.875445}, 'id': 0}, {'category': 'traffic light', 'attributes': {'occluded': False, 'truncated': False, 'trafficLightColor': 'green'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 1156.978645, 'y1': 136.637417, 'x2': 1191.50796, 'y2': 210.875443}, 'id': 1}, {'category': 'traffic sign', 'attributes': {'occluded': False, 'truncated': False, 'trafficLightColor': 'none'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 1101.731743, 'y1': 211.122087, 'x2': 1170.79037, 'y2': 233.566141}, 'id': 2}, {'category': 'traffic sign', 'attributes': {'occluded': False, 'truncated': True, 'trafficLightColor': 'none'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 0, 'y1': 0.246631, 'x2': 100.381647, 'y2': 122.825696}, 'id': 3}, {'category': 'car', 'attributes': {'occluded': False, 'truncated': False, 'trafficLightColor': 'none'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 45.240919, 'y1': 254.530367, 'x2': 357.805838, 'y2': 487.906215}, 'id': 4}, {'category': 'car', 'attributes': {'occluded': False, 'truncated': False, 'trafficLightColor': 'none'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 507.82755, 'y1': 221.727518, 'x2': 908.367588, 'y2': 442.715126}, 'id': 5}, {'category': 'traffic sign', 'attributes': {'occluded': False, 'truncated': True, 'trafficLightColor': 'none'}, 'manualShape': True, 'manualAttributes': True, 'box2d': {'x1': 0.156955, 'y1': 0.809282, 'x2': 102.417429, 'y2': 133.411856}, 'id': 6}, {'category': 'drivable area', 'attributes': {'areaType': 'direct'}, 'manualShape': True, 'manualAttributes': True, 'poly2d': [{'vertices': [[1280.195648, 626.372529], [1280.195648, 371.830705], [927.081254, 366.839689], [872.180076, 427.979637], [658.814135, 450.439209], [585.196646, 426.731883], [0, 517.817928], [0, 602.665203], [497.853863, 540.2775], [927.081254, 571.471352], [1280.195648, 626.372529]], 'types': 'LLLLLLLLCCC', 'closed': True}], 'id': 7}, {'category': 'lane', 'attributes': {'laneDirection': 'parallel', 'laneStyle': 'solid', 'laneType': 'road curb'}, 'manualShape': True, 'manualAttributes': True, 'poly2d': [{'vertices': [[503.674413, 373.137193], [357.797732, 374.672737]], 'types': 'LL', 'closed': False}], 'id': 8}, {'category': 'lane', 'attributes': {'laneDirection': 'parallel', 'laneStyle': 'solid', 'laneType': 'road curb'}, 'manualShape': True, 'manualAttributes': True, 'poly2d': [{'vertices': [[62.973282, 371.601649], [0, 368.53056]], 'types': 'LL', 'closed': False}], 'id': 9}, {'category': 'lane', 'attributes': {'laneDirection': 'parallel', 'laneStyle': 'solid', 'laneType': 'road curb'}, 'manualShape': True, 'manualAttributes': True, 'poly2d': [{'vertices': [[1274.517503, 376.208281], [905.986941, 371.601649]], 'types': 'LL', 'closed': False}], 'id': 10}]}

When downloading the BDD dataset
in bdd/labels/
I have renamed:
bdd100k_labels_images_train.json into train.json
bdd100k_labels_images_val.json into val.json

I would like to know how to fix this error

Thanks!

Thank you for your open source work.
For apply_net.py, the output json file include score, cls_prob, bbox_covar, etc.. But where's probability...
I'm looking forward for your reply.

Hi guys, thanks for your code! In order to reproduce your results, I would like to ask you to update your requirements.txt file by the specific version of detectron2 that you have used. This is very important since there are major changes in Detectron2 core functionalities from version 0.1.3 up to 0.4. Thank you!

Hi,

I want to be replicate your work without training the models for now. Do you have any of the per-trained models available?

Hi @asharakeh, thanks for sharing your work! According to your paper, the loss with bbox_cov prediction should be

but the implementation below seems to be :

It seems they are not equal. Am I miss something?

                    loss_box_reg = 0.5 * torch.exp(-pred_bbox_cov) * smooth_l1_loss(
                        pred_anchor_deltas,
                        gt_anchors_deltas,
                        beta=self.smooth_l1_beta)

                    loss_covariance_regularize = 0.5 * pred_bbox_cov
                    loss_box_reg += loss_covariance_regularize

                    loss_box_reg = torch.sum(
                        loss_box_reg) / max(1, self.loss_normalizer)

First of all, thank you for sharing your code.

I would like to ask your ideas for two issues I have met.
1. While I was working on running your evaluation (apply_net.py) with the method: BayesOD+Dropout, I have encountered the following error.

It says self.model has no attribute named 'in_features'. While debugging, I found that self.model has an attribute 'head_in_features' instead. Is it okay to replace 'in_features' by 'head_in_features' ?

The command I used is:
python src/apply_net.py --dataset-dir /My/BDD/Dir --test-dataset bdd_val --config-file BDD-Detection/retinanet/retinanet_R_50_FPN_1x_reg_cls_var_dropout.yaml --inference-config Inference/bayes_od_mc_dropout.yaml with your pre-trained model(retinanet_R_50_FPN_1x_reg_cls_var_dropout.pth)

2. When I run the command above, I get the notification as below:

It states that the checkpoint contains some non-used keys or missing keys. Does this indicate that I am using wrong checkpoint file?
The method I wanted to evaluate is BayesOD+Dropout, and I used the checkpoint (retinanet_R_50_FPN_1x_reg_cls_var_dropout.pth).

Thank you for reading.