saeed771 / coco-analyze

A wrapper of the COCOeval class for extended keypoint error estimation analysis.

coco-analyze Repository

This repository contains the code release from the paper Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation.

If you find this work useful please cite our paper:

@InProceedings{Ronchi_2017_ICCV,
author = {Ronchi, Matteo Ruggero and Perona, Pietro},
title = {Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {Oct},
year = {2017}
}

Important Content:

• pycocotools/COCOanalyze.py: wrapper of the COCOeval class for multi-instance keypoint estimation error analysis.
• COCOanalyze_demo.ipynb: ipython notebook showing how to use COCOanalyze as a standalone class.
• analysisAPI: API using COCOanalyze for an extended analysis.
• run_analysis.py: script generating a pdf summary of the extended analysis.

Installation

Use the Makefile to install the coco-analyze api:

• make all will compile and install locally. (RECOMMENDED)
• make install will install the api to the Python site-packages folder. NOTE This might override your current pycocotools installation.

Usage

To run the extended multi-instance keypoint estimation error analysis: update the paths of the detections and annotations and execute the command line.

[annFile]  -> ./annotations/keypoints_val2014.json
[dtsFile]  -> ./detections/fakekeypoints100_keypoints_val2014_results.json
[saveDir]  -> ./results/fakekeypoints100
[teamName] -> fakekeypoints100
[version]  -> 1.0
$ python run_analysis.py [annFile] [dtsFile] [saveDir] [teamName] [version]

Results

• A summary file called [teamName]_performance_report.tex will be created once the analysis is complete.
• All the generated plots are stored using [saveDir] as the base directory.
• Additional std_output information regarding the analysis can be found in the text files named std_out.txt.

Automatically Generated Performance Reports

You can find examples of the reports generated by the analysis code:

• Mask-RCNN
• CMU
• GRMI

Notes:

• The ./pycocotools/COCOeval class contained in this repository is a modified version of the original mscoco COCOeval class.
• The duration of the full analysis depends on the number of detections and size of the ground-truth split.
• You can comment out parts of run_analysis.py to run the analysis only for specific types of error.
• Set USE_VISIBILITY_FOR_PLOTS=True in localizationErrors.py if during the analysis you wish to visualize only the keypoints whos visibility flag is 1 (visible but occluded), or 2 (visible). Check issue #14 for more details.