mitmedialab / ajl.ai Goto Github PK

Add concept of consensus criteria to attribute/annotation types.

Consume this in postAnnotation controller (which is currently being hard coded to do 3 other annotatiosn 100% match)

This is a tool for image annotation, not face annotation.

I believe it to be an artifact of the LFW sample data set, but I'd like to propose we use a unique ID for each record instead of a name

opening this issue to discuss how state should be modeled.

I'm leaning toward remodeling the client side state, but perhaps both server and client should evolve toward each other now that we know more about this software.

I think this is the endpoint format @gnarf suggested in #81 / #85 that made the most sense -- we should update the backend to provide the data using consistent language to what was implemented in #85

• HTTP errors

annotation_type <-- top level names of all the annotation types
annotation_option <-- ethnicity/gender
annotation_range <-- age
annotation_box <-- regions
annotation_coord <--landmarks

Splitting out from #53: We should require users to have filled out an option for each step of the perceived demographics form before proceeding so they can't get the form into an invisibly invalid state.

This may be a good opportunity to add one final "confirmation" step before submitting & moving on to the next image, and/or to make the checkboxes auto-advance when selected.

This may mean moving form state into redux, per @gnarf's earlier feedback.

Either following the https://github.com/AJL-U/openfaces stack (angular 2) or a different one.

We added new API actions, but did not update the loading indicator reducers to account for them. This should be cleaned up.

administrate demographics

https://invis.io/HKACANH6Z#/217941303_Labeling_Demographics_-_Dawn_Pinch

administrate regions

https://invis.io/HKACANH6Z#/217941305_Labeling_Region_-_Dawn_Pinch

administrate landmarks

https://invis.io/HKACANH6Z#/217941304_Labeling_Landmarks_-_Dawn_Pinch

The boilerplate I am using for #4 does not have a built-in unit testing framework; this issue tracks pulling something like Jest into the project

Create a table for seed_truth and consensus_truth.

Several sample data faces are missing full landmark data, this issue tracks manually updating the sample data to add in that information

https://invis.io/HKACANH6Z#/218160452_Landmarks_-_Mobile

Specifically, start by implement a public_identities table and a link table with all of the images in the system.

identities:

• id
• real name
• public name
• imdb_id

Joy's model from mongo

• person_id (IMDB Unique Number)
• person_realname
• person_publicName
• dataset_enrollments (list of datasets with celebrity)
• Imgs_of_celebrities (list of img urls with celebrity)

Criteria:

• Install React Storybook
• Configure Storybook to work with our components
• (WIP)Add "stories" for existing presentational components
• Evaluate existing interface to evaluate what other UI should be moved into presentational components
• Add "stories" for main UI flow

https://invis.io/HKACANH6Z#/221452966_Welcome_-_Admin_Lady_Justice

We should implement a front-end router for the React application, and determine what the server infrastructure will be: should this be purely static w/ front-end routing, or have a server, etc

We should convert the existing fake api into something that fires an actual AJAX call to load the data, so that there will be an easier migration path once we have a read-write API to call against.

https://invis.io/HKACANH6Z#/218160455_Region_-_Mobile

Such as,

• You've finished a workload & passed validation - take a moment to call that out, then here's the next workload
• You've finished a workload and your answers do not match the "truths" - call that out and give them ability to re-start w/ the new workload
• You just tried to annotate a workload that was not sent to you
• You have been "enrolled"

Some of the errors (see #53) are more a "state" that the user is in, rather than a system error; we should expand whatever we do for error handling, to account for these workflow actions

either by pointing it at a remote test db, or at a local moch data server.

In addition to the commits on PR #78, create logic to set a new truth in the known based on some arbitrary criteria of a "annotatable attribute". Perhaps the first pass is '4 different annotators agree'.

Define the data model for the application state and structure it using Redux

Depends on #4

When a session posts it's first set of 12 annotations

• check if they got the 8 ground truths correct
• if they didn't get 8 right then have them start over
• if they did, then check if the 4 unknown images have 4 agreeing submissions and make them into new ground truths
• then respond with a new 3 image workload

per @gnarf eg: compute accuracy

For example:

a grid of 12 images where you select all the people who have the same gender as you.

CC @joyab Can you say more about this?

• Remove skintone from db scaffold and replace with ethniciy
• Add endpoint for bootstrapping the client annotation types:

// to load the annotation types into the client (eg /api/v1/annotation_types
  annotations:{
    demographics: [{
        name: 'Perceived Age',
        options: ['infant', 'child', 'young adult', 'adult', 'elderly']
      },
      {
        name: 'Perceived Gender',
        options: ['other', 'female', 'male'],
      },
      {
        name: 'Perceived Ethnicity',
        options: ['balck', 'white', 'lantino/a', 'asian', 'other']
      }
    ],
    regions: [{}],
    landmarks: [{}]
  },

• Add endpoint for bootstrapping the client work loads:

// to load workloads for the client (eg /api/v1/work)
// this would load 12 work items to start, followed by 3 each
// question about whether the server should default to knowing how many to send based on session, 
// or if this should be specified with a query param 
  workload : [{},{},{},{},{},{},{},{},{},{}]

• Add endpoint for accepting annotations from the client:

// to take an array of annotations to be processed (eg /api/v1/annotations)
var annotations = [
  {
    image_id: ''
    demographics:[{
      name: '',
      option: ''
    },{
      name: '',
      option: ''
    }],
    regions: [],
    landmarks: [{}, {}, {}],
  },
  {...}
]

per @cowboy

so people don't have to get postgres running.

• visualize annotation stats from annotation campaign
• visualize ethnic/gender/age of landmark-annotated faces

Primary goal: motivate citizens to participate.

Show users how their participation improves their participation

Pulled out from @gnarf's review of #65: we have a comment that says,

This post must contain only and all annotations for images sent to the current session. If there is a discrepancy between the current workload sent to the current session's annotation post, the post will return an error 500.

We should probably return a 400 "Bad Request" error instead

• Perceived ethnicity: https://invis.io/HKACANH6Z#/218146722_Perceived_Ethnicity_-_Mobile
• Perceived age: https://invis.io/HKACANH6Z#/218146720_Perceived_Age_-_Mobile
• Perceived gender: https://invis.io/HKACANH6Z#/218146724_Perceived_Gender_-_Mobile