newsdev / who-the-hill

Who The Hill: An MMS-based facial recognition service for members of Congress.

Who The Hill

What is Who The Hill?

Shazam, but for House members faces.

Who The Hill is an MMS-based facial recognition service for members of Congress. Reporters covering Congress can text pictures of members of Congress to a number we’ve set up and they’ll get back:

1. A list of all the members of Congress recognized in their picture
2. Numbers indicating how confident Amazon is about the recognition
3. Colors corresponding to each member of Congress recognized
4. The picture they sent, but with a box around each member of Congress in the color corresponding to them

Local Installation

Before getting the app running, you'll need a Twilio account (with an operational MMS number), an Amazon Rekognition account and an Amazon S3 or Google Cloud Storage account.

Who The Hill also requires Python3 and works best with virtualenv and virtualenvwrapper. For more on how NYT Interactive News sets up our Python environment, check out this blog post by Sara Simon.

git clone https://github.com/newsdev/who-the-hill.git && cd who-the-hill
mkvirtualenv whothehill
pip install -r requirements.txt

To run the app locally, you will need some environment variables.

Three sets of AWS (or AWS-like) keys are needed:

• AWS Rekognition credentials

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION

• AWS S3-like service endpoint and credentials

AWS_S3_ENDPOINT
AWS_GCS_ACCESS_KEY_ID
AWS_GCS_SECRET_ACCESS_KEY

(You can find more information about the AWS and AWS-like credentials here.)

You will also need credentials and a number from Twilio.:

TWILIO_ACCOUNT_SID
TWILIO_AUTH_TOKEN
TWILIO_NUMBER

To check whether results returned from Rekognition are actually members of Congress, you can either use the json dump of members of Congress (as well as variations on spellings of their name) included in this repo, or use your own API endpoint that returns similarly formatted json. If you don't set this environment variable, Who The Hill will default to using the included json file:

NICKNAMES_ENDPOINT

You can store your environment variables in a dev.env file...

export AWS_ACCESS_KEY_ID='<YOUR_ID>'
export AWS_SECRET_ACCESS_KEY='<YOUR_ACCESS_KEY>'
export AWS_DEFAULT_REGION='<YOUR_PREFERRED_REGION>'
export TWILIO_ACCOUNT_SID='<YOUR_ACCOUNT_SID>'
export TWILIO_AUTH_TOKEN='<YOUR_AUTH_TOKEN>'
...

...and run source dev.env. This will export your credentials to your environment.

Running the App

You can run the app locally as a web service that integrates with Twilio or as a CLI for examining a folder full of images to recognize.

As a web application

Run the app locally python who_the_hill/web/pub.py and tunnel with ngrok so that you can integrate with Twilio, which needs a public-facing endpoint to POST data to.

As a CLI for recognition

Put the images within which you'd like to recognize members of Congress into a folder like /tmp/to_recognize and then call the CLI like this:

python who_the_hill/cli --directory /tmp/to_recognize/

The app will examine the images, find and recognize faces, and produce a JSON report. Note: The CLI still requires working S3/GCS tokens and (obviously) access to the AWS Rekognition API. It does not require Twilio credentials, though.

Acknowledgements

Jennifer Steinhauer came up with the original idea behind Who The Hill and was an enthusiastic sponsor and tester.

Who The Hill was developed by Interactive News interns Gautam Hathi and Sherman Hewitt in the summer of 2017 and partially rewritten in the spring of 2018 by Jeremy Bowers, all under the watchful eye of Rachel Shorey.

who-the-hill's Issues

Each lawmaker gets a separate text in return

Running into 140-character issues with groups of 3 or more.

Add party and state info to results

Perhaps also then return each member with an individual text to avoid awkward cutoffs?

Unit test coverage

Because tests are important, just not important enough for us to have written them before now

Specifying how to make S3 bucket public, read-only

When trying to implement this locally, I ran into the issue of only getting the text with names back. No image was returned.

Some snooping led me to the following Bucket Policy JSON:

{
  "Version": "2008-10-17",
  "Statement": [{
    "Sid": "AllowPublicRead",
    "Effect": "Allow",
    "Principal": { "AWS": "*" },
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::who-the-hill/*" ]
  }]
}

This snippet makes it so your bucket is public and read-only, thus allowing Twilio to access the uploaded image and send it in a response. In addition, I granted List privilege to Everyone in the Permissions section of the bucket.

Would it be worth specifying this in the documentation?

NYT-specific links left in code

A leftover NYT link caused some issues with receiving replies from Twilio. I address this in PR #6

Indentation Error and undefined names

flake8 testing of https://github.com/newsdev/who-the-hill on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./who_the_hill/rek/models.py:182:21: F821 undefined name 'recognized_faces'
        for face in recognized_faces:
                    ^
./who_the_hill/rek/utils.py:10:4: E999 IndentationError: unexpected indent
    if on_after_date is not None
   ^
./who_the_hill/web/pub.py:99:12: F821 undefined name 'alerter'
MY_ALERT = alerter.Alerter(counter_limit=100, time_interval=3600, recipients=getRecipients())
           ^
1     E999 IndentationError: unexpected indent
2     F821 undefined name 'recognized_faces'
3

No "failure message" for non-Congresspersons

twilio_app.py runs into this error when non-Congresspersons are sent in:

Traceback (most recent call last):
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\_compat.py", line 33, in reraise
    raise value
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1614, in full_dispatch_reque
st
    rv = self.handle_user_exception(e)
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1517, in handle_user_excepti
on
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\_compat.py", line 33, in reraise
    raise value
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1612, in full_dispatch_reque
st
    rv = self.dispatch_request()
  File "C:\Users\Sherman\DOCUME~1\PROGRA~1\Github\WHO_TH~1\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\Sherman\Documents\Programming Projects\Github\who_the_hill\twilio_app.py", line 170, in recongize
    del target_image
UnboundLocalError: local variable 'target_image' referenced before assignment

...which results in no response from Twilio. Congresspeople do elicit a Twilio response, as expected. Images without recognized faces trigger a "failure message," as expected.

Example: Send in picture of Marco Rubio, get back Marco Rubio and image.
Example: Send in picture of a chair, get back failure message.
Example: Send in picture of Will Smith, get back nothing.

missing requirements.txt

Fuzz the exposed phone number in the README example image

Even if it is no longer used, if it was provided by twilio, it may be recycled in the future.
You should fuzz it so that no future user is inexplicably plagued by unsolicited pictures of Mitch McConnell.