Ground truth eye locations with COTS plugins about openbr

[caotto]

So at least the CT8 plugin can already use external eye locations, but what
I would like is for all COTS plugins to have the ability to use external
eye locations (could be ground truth, could be the output of another
detector) if available, and to do that in a standardized way.

I don't necessarily know the right way to standardize it--there are
currently two ways to store points, as either anonymous points in a list,
or as a separate key/value pair for each point (named points). The CT8
plugin assumes the eye locations are stored as the first two anonymous
points of a template and doesn't consider named points, which isn't
particularly explicit.

There has been some discussion on standardizing point names, and some
changes which I haven't really followed so maybe @jklontz can expand or
disagree, but what I would like is for the two eye locations to be
consistently stored as named points with some standard name, and for the
COTS plugins to use those if available. Another option would be for the
transforms to take key names of the eye locations as arguments (so you
could store both ground truth, and detected points as named points, and
selectively use either for enrollment).

Anyway in general I think it would be better to modify existing transforms
to optionally accept eye locations for enrollment (where feasible) rather
than making new transforms.

-Charles

On Sat, May 11, 2013 at 3:34 PM, Brendan K [email protected] wrote:

@jklontz https://github.com/jklontz @caotto https://github.com/caottoI am interested in using a COTS plugin to match face images in a modality
where face and eye detection generally fail. As such, I want to provide
ground truth eye locations in the sigsets, and use these values at the
enrollment stage for the COTS matcher.

From examining the COTS plugin code, this operation does not seem to be
supported in openbr. Further, it would seem the fix would be to create a
new plugin for each corresponding COTS matcher, which uses the meta data
provided in the sigset (i.e., eye locations) to enroll the face image. Does
these two statements seem correct? If so, then I will likely create at
least one such plugin.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/51
.

[jklontz]

I agree with everything that's been said here, let me just fill in some details about how I think it should work.

Step 0 is that for a given COTS matcher, their face detector and template generator should be separated into two distinct transforms. CT8 does this already with CT8Detect and CT8Enroll. I'm not sure about the status of the other systems off the top of my head, but the only one where I know this is impossible is PP5.

As Charles suggested, the other step is to decide on standardized names for the landmarks returned by the detector so that they can be specified from a sigset. I think firstEye, secondEye, noseTip, or something along those lines are good candidates for names. These standardized names should be added to the br::File documentation which maintains a list of standardized metadata fields. On a related note, I'd like to stop storing these points anonymously as we do now with both ASEFEyes and CT8Detect, and reserve the anonymous storage for points that truly have no meaningful label, like SIFT keypoints.

Sound good?

[bklare-zz]

Thank you both for the detailed replies. Every point you all bring up makes
sense, and, from an experimental standpoint, it seems there would be value
to separating these stages.

Unfortunately, the one matcher I was concerned at this point is PP5. B/c it
is apparently not possible to do manual enrollment with PP5 (as Josh
noted), this is issue is a no longer a personal priority.

Would it make more sense to use the craniofacial landmark names instead of
firstEye, noseTip, etc? On the plus side this may help towards consistency
amongst different landmark extractors. On the down side, these are
difficult to learn.

On Sun, May 12, 2013 at 9:46 AM, jklontz [email protected] wrote:

I agree with everything that's been said here, let me just fill in some
details about how I think it should work.

Step 0 is that for a given COTS matcher, their face detector and template
generator should be separated into two distinct transforms. CT8 does this
already with CT8Detect and CT8Enroll. I'm not sure about the status of the
other systems off the top of my head, but the only one where I know this is
impossible is PP5.

As Charles suggested, the other step is to decide on standardized names
for the landmarks returned by the detector so that they can be specified
from a sigset. I think firstEye, secondEye, noseTip, or something along
those lines are good candidates for names. These standardized names should
be added to the br::File documentation which maintains a list of
standardized metadata fields. On a related note, I'd like to stop storing
these points anonymously as we do now with both ASEFEyes and CT8Detect, and
reserve the anonymous storage for points that truly have no meaningful
label, like SIFT keypoints.

Sound good?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/51#issuecomment-17778154
.

[jklontz]

@bklare Yeah I'd certainly be willing to consider craniofacial landmark names, and I agree with the strengths and weaknesses you've stated. I'm going to close this ticket for the time being based on your response, we can re-open when someone renews interest in this area.