Detection Analysis about openbr

[jklontz]

Totally agree. We currently have a few transforms in the grammar for ad hoc visualization. For example:

$ br -algorithm "Open+Cascade(FrontalFace)+ASEFEyes+Draw+Show" -enroll data/MEDS/img/S001-01-t10_01.jpg

There is also the EditTransform for manually editing landmarks but I think it currently only supports deleting them :) You can find all the relevant code in sdk/plugins/draw.cpp and sdk/plugins/misc.cpp. Currently no real GUI support otherwise. If you're ready to start implementing more quantitative feedback, let me know and I can suggest a couple ways to implement it.

[jklontz]

I'm finally ready to take on this ticket. The approach I'm proposing is to introduce br -evalDetection and br -evalLandmarking that act similarly to br -evalClassification and br -evalRegression in that they take an a file of ground truth detections/landmarkings and a file of computed detection/landmarkings and print the error statistics to the terminal. Sound good?

[bklare-zz]

@jklontz Great to hear. I agree with the proposed interface.

I suppose the next decision will be the format with which to represent annotated faces and landmark sets, where annotations could be manual/ground truth, or automated.

[jklontz]

As far as formats go, we have free support for all the existing Gallery formats (including .xml and .csv). If we need any other formats we can simply add new plugins. I'll take a look at what you do right now in your java code and see if it works out of the box, otherwise I'll see if I can find a way to support it, or in the worst case change it.

Want to weigh in on what error metrics to support?

[bklare-zz]

Face detector metrics:
-Percent of faces correctly detected (requires an overlap parameter)
-Number of false positives
Notes: Classic Type I and Type II error analysis is difficult here b/c the
every single sliding window location is essentially a classification stat.
Obviously this dilutes the results. Thus, we can still measure Type II
error (i.e., rate of false negatives), but the Type I error (i..e, rate of
false positives) is instead reported as the "number" (not rate) of false
positives. Overlap parameter is what percentage two bounding boxes need to
overlap to be considered a match (e.g., 75%).

Landmark detector metrics:
-Total IPD normalized mean landmark distance
-Per landmark, IPD normalized mean landmark distance
Notes: This is also referred to as the 'bi-ocular' distance metric. See
Section 5.1 from the paper below for a further description.

Sun, Yi, Xiaogang Wang, and Xiaoou Tang. "Deep Convolutional Network
Cascade for Facial Point Detection." IEEE CVPR 2013
http://www.ee.cuhk.edu.hk/~xgwang/papers/sunWTcvpr13.pdf

On Tue, Jul 23, 2013 at 12:37 PM, jklontz [email protected] wrote:

As far as formats go, we have free support for all the existing Gallery
formats (including .xml and .csv). If we need any other formats we can
simply add new plugins. I'll take a look at what you do right now in your
java code and see if it works out of the box, otherwise I'll see if I can
find a way to support it, or in the worst case change it.

Want to weigh in on what error metrics to support?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-21427493
.

[caotto]

What do you think about evaluation detection performance using a precision/recall curve

[bklare-zz]

This is a good suggestion. Precision/recall is not (generally) sensitive to the fact that each sliding window location is a classification instance b/c it only uses TP, FP, and FN (i.e,. it does not use TN). My only downside is that I often find precision/recall hard to read from a semantic interpretation standpoint, but this may just be may.

You inadvertently bring up a good point about the that our evaluation should support a continuous/thresholdable output. The popular FDDB benchmark uses a modified ROC curve, where instead of false positive rate, it is total number of false positives (again, b/c each location is a classification point). Below is an example of this plot.

Perhaps we could use precision/recall and the modified ROC plot as our metrics for (face) detection.

http://vis-www.cs.umass.edu/fddb/results.html

[jklontz]

It sounds like what we want, at least for evaluating detection performance, is a setup similar to how we evaluate recognition:

1. Generate an intermediate results .csv file for a detector, and print out a few numeric statistics to the terminal.
2. Plot one or more results files together to compare detector performance.

The three plots that come to mind immediately are:
1/2. FDDB's false positive count vs. true positive rate (both continuous and discrete versions).
3. Histogram of detection bounding box overlap percentage

Sound good?

[bklare-zz]

Yes, sounds good. Down the road, we can use the CSV to generate precision
recall curves should we want to present these stats later.
On Jul 25, 2013 10:31 AM, "jklontz" [email protected] wrote:

It sounds like what we want, at least for evaluating detection
performance, is a setup similar to how we evaluate recognition:

1. Generate an intermediate results .csv file for a detector, and print
   out a few numeric statistics to the terminal.
2. Plot one or more results files together to compare detector performance.

The three plots that come to mind immediately are:
1/2. FDDB's false positive count vs. true positive rate (both continuous
and discrete versions).
3. Histogram of detection bounding box overlap percentage

Sound good?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-21557725
.

[jklontz]

Oh sorry, forgot about that curve -- I'll add it in too.

[jklontz]

The detection half of this ticket has been implemented and described on the mailing list.

[bklare-zz]

@jklontz Thanks for getting this working. Everything looks really good so far.

Per our discussion, we will also need to ingest FDDB files in the 'ellipse' format. In this format, a face entry is specified as:
major_axis_radius minor_axis_radius angle center_x center_y confidence

To maintain consistency with our other (Java) code, we will want to convert this to a rect in the following manner:
rect.x = center_x
rect.y = center_y
rect.width = minor_axis * 2
rect.height = rect.width

If anyone has an issue with this conversation just let me know. It will not be difficult to change our other code if this is the case.

[jklontz]

@bklare Can you weigh in a bit more on how you'd like to see the landmarking accuracy plots? I know we talked about plotting % of data samples with error under a certain fraction of IPD. My concern is that we have too many landmarks to allow for meaningful comparison between different landmarks and different algorithms on the same figure. I think we can get all the important information on a single plot using boxplots, assigning different colors to different algorithms, and faceting by landmark location. Also, do you envision each landmark to be individually named or simply numbered from 1 to n?

[bklare-zz]

@jklontz Regarding the plots for percent of samples with accuracy under a
certain (normalized) distance, I think this should be reflective of the
mean distance for all landmarks. This way there is only one plot. While one
critique on this approach is that it may be dirtied by have too many
landmarks, I would think such cases mean that the underlying protocol needs
to be changed (e.g., conduct experiments on smaller subsets of more
meaningful landmarks). Does what I am saying make sense, and, if so, do you
agree?

Regarding names, I think simple numbering is fine. The only case where I
could see this changing is if we wanted to use anthropometric landmark
naming. That can be dealt with at a later date should the need arise.

On Wed, Aug 7, 2013 at 3:08 PM, jklontz [email protected] wrote:

@bklare https://github.com/bklare Can you weigh in a bit more on how
you'd like to see the landmarking accuracy plots? I know we talked about
plotting % of data samples with error under a certain fraction of IPD. My
concern is that we have too many landmarks to allow for meaningful
comparison between different landmarks and different algorithms on the same
figure. I think we can get all the important information on a single plot
using boxplots http://docs.ggplot2.org/current/geom_boxplot.html,
assigning different colors to different algorithms, and faceting by
landmark location. Also, do you envision each landmark to be individually
named or simply numbered from 1 to n?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-22275729
.

[jklontz]

@bklare I hadn't considered this mean approach, though I agree with your justification. I'll implement both this and the boxplot scheme I proposed.

[bklare-zz]

Great. And good call on the box plot.

On Wed, Aug 7, 2013 at 5:16 PM, jklontz [email protected] wrote:

@bklare https://github.com/bklare I hadn't considered this mean
approach, though I agree with your justification. I'll implement both this
and the boxplot scheme I proposed.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-22284721
.

[jklontz]

Finished implementing in 4c3ee28, and described in https://groups.google.com/forum/#!topic/openbr-dev/dOdkcD4YL0E . Finally done with this :)