sc3 / cookcountyjail

A Django app that tracks the population of Cook County Jail over time and summarizes trends.

What is the Supreme Chi-Town Coding Crew?

We are a small group of aspiring web developers and journalists based in Chicago who meet regularly to learn basic web skills and work on data journalism and civic technology projects. Learn about SC3's principles in our manifesto.

How do I join?

Come to OpenHack at FreeGeek Chicago, every Saturday from 1pm-5pm at 3411 W. Diversey. Details available on the FreeGeek Chicago calendar.

Projects

• Cook County Inmate Tracker - A Django-based web API that provides data about inmates in Cook County Jail by scraping for the Sheriff's Inmate Locator. Status: 2.0 version in development. 1.0 version is inefficient and complicated but stable. Both versions are available online.
• 26th and California - Tarbell-based data visualizaton site for the Cook County Jail API. Status: In active development.
• Townsquare 2 - A Django application for tracking volunteer participation. Status: Alpha quality, in development.
• Chicago Birthrates - A simple application that visualizes birthrates in Chicago based on the city data portal. Status: Stable version 1.0, demo site is broken.
• Townsquare 1 - A Drupal distribution for tracking volunteer participation. Status: In production at FreeGeek Chicago, not maintained as a public project.
• Hopper VM - A lightweight virtual machine for apps development Status: Not maintained.

---

License

All files in this repository are by the work by Supreme Chi-Town Coding Crew and licensed under a Creative Commons Attribution 4.0 International License.

cookcountyjail's Issues

Add parameter to filter by booking date to scraper functions.

To make scraping quicker, we need to filter for inmate records with booking date greater than some parameter, and also calculate discharge dates against the same set of records.

The handle, process_urls, and calculate_discharge_date methods in scrape_inmates.py will need to get this parameter. handle needs to get it from the command line arguments, and then pass it as a function parameter to the other two functions.

Get Cook County Jail floorplan

When an inmate record is update, the last_seen field should be set to current datetime

Last_seen is only set when an inmate record is created. Maybe we want to keep this info, but it should be called first_seen.

Implement TastyPie

Research other data sources

Add housing location history to data model

Court Date endpoint should be filterable by date

To replicate

1. Open http://cookcountyjail.recoveredfactory.net/api/1.0/courtdate/?format=json&date=2012-03-17

Expected Result

Retrieve a list of court date objects filter to only show the ones for March 17, 2012

Observed Result

An error message that reads The 'date' field does not allow filtering.

Create south data migration for new housing location model

Create fabfile

Fairly low priority at the moment -- you should really just look into what fabric is and what it does. We need something, eventually, that can at the least bootstrap the database.

Cover relationships to court location with tastypie

Add court date history model, relate it to inmate model

Update poll_mode to new create_update_inmate function

Integrate South?

I kind of hate south. But we probably need it because we're scraping data.

Add housing_history to API

Add full housing_history functionality to the API. Have the reverse relationships with the housing_locations displayed like court_dates.

Court date parsing is borked

@wilbertomorales and I made a little mistake in our refactoring.

Housing Location discrepancy

Looking at the API's JSON output and comparing it to the search page, every piece of data is correct except housing_location. @wilbertomorales and I wrote a single inmate scraper to pull one inmates data and compare it to the JSON output, and the data is scraped correctly.

So... this happens somewhere between the database and output. Anyone have any idea?

Add court date API documentation

Who wants this? The interface works the same as locations and inmates, see http://cookcountyjail.recoveredfactory.net/api/1.0/courtdate/?format=json and http://cookcountyjail.recoveredfactory.net/api/1.0/courtdate/?format=json&related=1

Generate summaries of all distinct places

Should be as simple as querying API for CourtLocations.

"Poll" mode for inmate scraper

The current scraper process assumes a long, full-blown search of all inmates at once-a-day intervals. But because of the possibility of short-term prison stays, we need some faster, less bandwidth intensive ways of scraping.

I suspect a mix of two techniques could help:

1. Polling regularly for new inmates using the last known jail id #. Right now, that's http://www2.cookcountysheriff.org/search2/details.asp?jailnumber=2012-1128234 -- if you polled every X minutes for 2012-1128235, you could quickly pick up new inmates at they're added.
2. Searching the last X days' worth of inmates every hour or two to look for recent dropouts. The value of X should be whatever duration we think hour-level precision for time of discharge matters. For long stays, precise discharge time becomes less important, and so can be excluded.

I'm curious if these should be their own management commands or should be specified by a flag to scrape_inmates.

Add Discharge date for poll_mode

Refactor process_urls

Separate the process_urls and the field parsing.

Create south schema migration for new housing location models

Let's research Tastypie

@Davy-jones-Giant-squid @ryandasher @codersquid @wilbertomorales @gloverjimmie -- please take a gander at http://tastypieapi.org/ and http://django-tastypie.readthedocs.org/en/latest/

Set up new repo for user interface, clean up this repo

The plan for the January class should be to remove UI stuff from this repository and move it to another project, probably called "fullhouse", which will be ridiculously simple: A static HTML template that anyone with git can clone and work on. Here's a general layout to start from:

• index.html: Load the javascript, build up the app
• js/
  • js/config.js: Global configuration variables: URL data API (defaults to fullhouse.recoveredfactory.net).
  • js/app.js: Backbone app.
• css/
  • css/style.css: Stylez.
• lib/: Librariez.

This will make environment issues essentially moot -- those who want to build out data viz side of project need a browser and git.

Pick a license for code (and also data?)

Parsing the location fields

We need to understand the location code in order to figure out whether a person is in jail, on electronic monitoring, awol, or otherwise disposed.

We might also want to do further parsing so that we could figure out which building and tier an inmate is in. This could be very nice for visualization purposes, but could also lead to insights about the internal operations of the jail.

These documents are a start: https://docs.google.com/spreadsheet/ccc?key=0AptDdMTTIKEndDdJUEhNMXhWWDhDMHVQejRWZEtoeFE https://docs.google.com/spreadsheet/ccc?key=0AptDdMTTIKEndGs3ZUdKTlpRdFVzMG1YRXFVbkVKeFE

But they don't completely square with how the Sheriff describes the jail: http://www.cookcountysheriff.org/doc/doc_DivisionsOfJail.html

For example is division 03-* the residential treatment unit or an auxillary?

Calculate BOOKING_AGE from Date of Birth and Booking date in Scraper

We want to store the person's age not the date of birth, which means we need to calculate the age from DOB in in scraper.

The right way to do this is to

1.) Subtract the DOB year from the BOOKING year. This gives BOOKING_AGE
2.) Check to see if the DOB day and month is after BOOKING day and month. If it is after, decrement BOOKING_AGE by one.

Parse Court Locations

It might be a good idea to start parsing the court locations with more detail. The objects currently look like this:
{
"id": "1",
"location": "Criminal C\nCriminal Courts Building, Room:506\n2650 South California Avenue Room: 506\nChicago, IL 60608\n"
},
If we break up the different segments in the models it will provide a better API. For example a model with:

location_name = Criminal C
branch_name = Criminal Courts Building
room_number = 506
address = 2650 South California Avenue,
city = Chicago,
state = IL
zip_code = 60608

Would make it possible to map out all the court dates currently happening at a specific location and, give us a better view of the traffic on it. Maybe not so far as to break up the city and state but far enough to separate it's zip code. With a zip code having it's own field we would be able to work at a much more abstract way.

Add all fields to model

Add all fields from a page like http://www2.cookcountysheriff.org/search2/details.asp?jailnumber=2012-1121092 to the InmateRecord model.

Use get_or_create the right way in scraper

Sketch out demo UI

Booking date is incorrectly being assigned to what is apparently "now"

Calculate discharge time

When processing, we'll probably want to compare the jail_ids that are harvested with jail_ids of people in the database without a discharge date. That should give us all the inmates who fell out of the system since that last time we ran the scraper. We can then assign an inferred discharge date to their record.

Look into csv output

Associate court locations with judges and extended location metadata

Determine and account for values that were causing next court date to throw errors in scraper

Too many locations

There's something funny going on with the locations.

I'm getting things like
| 683 | Maywood
Criminal Courts Building, Room:306
2650 South California Avenue Room: 306
Chicago, IL 60608
|

where the location_id 683 does NOT appear in the countyapi_courtdates table.
Right now about 280 distinct locations appear in the countyapi_courtdate table and
there are about 720 locations in the countyapi_courtlocations table.

Write tests

I need to brush up on writing Django tests.

Sketch out basic API

Set up server (EBS?)

I'm leaning towards Elastic Beanstalk on this, which supports Django + MySQL.

Charges fields are being saved empty

Add caching

See https://devcenter.heroku.com/articles/increasing-application-performance-with-http-cache-headers

Drop lastname, firstname fields

Scrape all fields from inmate pages and assign to inmate record

This is in countyapi/management/commands/scrape_inmates.py around line 90. You'll want to process and map specific columns to each InmateRecord field.

Fix Court Dates

Look into cross-correlating with other datasets

@bepetersn:

datasets to looks into: fugitive, prison, crime incidents, judges, geodata

Fix calculate_age fuction

It is sometimes calculating the wrong age

CSV output for courtdates

Through the API, should be able to request a csv of just courtdates. This should be in the form
of

jail_id, date, court_date_id, location_id

Run full scrape daily on Heroku

Generate Inmate Summaries

Generate Inmate Summaries

This is a requested feature and it is easy enough to implement since we already have a table and command for it. All we need to do is implement it. Something to keep in mind is that all summaries should be a day or more behind because the cook county website seems to be behind.