deardurham / ciprs-reader

Python library for reading CIPRS PDFs

CIPRS Reader

Setup and Run:

Add pdf file to parse in /ignore folder then run:

docker build -t ciprs-reader .
docker run --rm -v /$(pwd):/usr/src/app ciprs-reader python ciprs-reader.py ignore/cypress-example.pdf

Example output:

[
    {
        "General": {
            "County": "DURHAM",
            "File No": "00GR000000"
        },
        "Case Information": {
            "Case Status": "DISPOSED",
            "Offense Date": "2018-01-01T20:00:00"
        },
        "Defendant": {
            "Date of Birth/Estimated Age": "1990-01-01",
            "Name": "DOE,JON,BOJACK",
            "Race": "WHITE",
            "Sex": "MALE"
        },
        "District Court Offense Information": [
            {
                "Records": [
                    {
                        "Action": "CHARGED",
                        "Description": "SPEEDING(70 mph in a 50 mph zone)",
                        "Severity": "TRAFFIC",
                        "Law": "20-141(J1)"
                    }
                ],
                "Disposed On": "2010-01-01",
                "Disposition Method": "DISMISSAL WITHOUT LEAVE BY DA"
            }
        ],
        "Superior Court Offense Information": [],
    }
]

Local Setup

Pre-requisites:

Mac

brew cask install pdftotext

Ubuntu

sudo apt-get install -y poppler-utils

wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.04.tar.gz \
    && tar -xvf xpdf-tools-linux-4.04.tar.gz \
    && cp xpdf-tools-linux-4.04/bin64/pdftotext /usr/local/bin/pdftotext-4

Setup:

pip install -r requirements.txt
pip install -e .

Read CIPRS PDF:

python ciprs_reader.py ./cypress-example.pdf

Run Jupyter:

jupyter-lab

Run tests:

pytest --pylint

Code for Durham

ciprs-reader's Issues

Parsers do not catch numbered offense rows

The offense row parser is designed to capture data from offense rows that look like this:

ACTION DESCRIPTION SEVERITY LAW CODE

However, on some records they look like this:

# ACTION DESCRIPTION SEVERITY LAW

With # being either a zero padded number or blank

Offense date parsing fails with 00:00 time

Works:

Matched: {'value': '08/04/2004 01:00 AM'} in Offense Date/Time: 08/04/2004 01:00 AM                         • Date: 04/05/2005

Doesn't work:

No match: Offense Date/Time: 03/04/2005 00:00                            • Date: 02/16/2006

Offense Date is taken from Citation Info, which may not exist

Ideally we should be pulling the Offense Date from the Case Information section, which is the first section at the top. Right now the parser gets it from the citation information section, which doesn't exist for some records.

County names with spaces are not parsed properly

The CaseDetails parser doesn't expect county names to contain spaces, so a line like

Case Summary for Court Case: NEW HANOVER 20CR000000

will result in a county name of NEW.

AC

• CaseDetails parser updated to expect spaces
• A test is added wi spaces in the county name

OffenseRecordRow parser skips rows with descriptions containing slashes

Rows like the following are skipped currently because the regex fails to match:

CONVICTED        CITY/TOWN VIOLATION (I)                            INFRACTION                     LOCAL ORDINANCE

AC

• OffenseRecordRow updated to expect non word-characters
• Tests added for examples with slashes

Extract record type

A machine-readable record type will be useful for dear-petition.

AC:

• Add Record Type: Summary|Detail to JSON export

County names with special characters don't match

The CaseDetails parser fails to parse county names with special characters like:

No match: Case Summary for Court Case: GUILFORD-GR 15CR000000

The petition tool then groups all unmatched county names under UNKNOWN, which is a bug.

AC:

• Update CaseDetails to match non A-Z characters. Perhaps just match anything that's not a space? E.g. (?P<county>.+)
• Add test with special characters

Tracking issue for release v1.4.0

• #50

Removed unnecessary steps. We should release now to resolve Jessica's issue with Open Container description not getting parsed

Extract “additional offenses exist” language

Summary records can contain the following language:

Additional offenses exist. To see a complete breakdown, view full case detail.

If it exists, this should be noted in the JSON so dear-petition can pick it up and display a message to the user:

{
    "Additional offenses exist": "true"
}

Parsers do not catch multiple offense records.

Some CIPRS records look like this:

Offense Record 1 of 2 (Line Number 1)
...
Offense Record 2 of 2 (Line Number 2)
...

We are only capturing the charges from the first offense record and not the second.

Update date parsers to return MM/DD/YYYY format

Right now date parsers return datetime strings, which get converted to MM-DD-YYYY format for display, but this results in a lot of duplicated code because it always needs to be converted to MM/DD/YYYY (%m/%d/%Y) format in the end. We can DRY our code by making the parser return that format in the first place.

Name parser needs to be able to include non-letter characters

The name DEANS,TEST'TEST,TYNAHJ gets parsed as Shaqui Deans rather than Shaqui'awan Deans because it stops at the '.

Jurisdiction on infraction offenses should be District

The CIPRS reader only recognizes CR (DISTRICT COURT) and CRS (SUPERIOR COURT) values in the CIPRS record file numbers, causing the Summary Document to put the infraction offenses in a NOT AVAILABLE jurisdiction. The infraction offenses should be put in the DISTRICT jurisdiction.

For testing/demo: Can use CIPRS record of person with initials JR or MJ

Multi-Line Reader Mode on Mac with M1 processor

Make Offense Disposed On Date Parser work on Case Summaries

There are the two types of court records: Case Details and Case Summaries. The petition generator was built for the Case Details. Make this parser work for Case Summaries as well.

Extract Plea and Verdict offense fields

Export JSON should contain:

{
    "Offense Record": {
        "Plea": "blah",
        "Verdict": "blah",
    }
}

Parse multiple CIPRS records from combined PDF

Attorney's manually combine CIPRS record PDFs to ease organization of his documents, so a single PDF may contain many CIPRS records.

AC

• Split extracted PDF text on common delimiter. Potential options:
  • Maybe “Case Summary for Court”
  • Footer record number of page count
• Run entity extraction separately on each record
• Always return list of CIPRS records as JSON, even if majority will be single records
  • A continuation issue will need to be opened in dear-petition to always handle lists

Disposition method parser doesn't expect non-word characters

Disposition Method: WAIVER - MAGISTRATE is currently parsed as WAVIER.

AC

• OffenseDispositionMethod parser updated to expect non-word characters
• A test is added with dashes

Use disposition codes in offense section

The DEAR team uses disposition codes to fill in the disposition column in the offense section. Update petition generator to use these codes.
ACIS codes.pdf