dismantl / caseharvester Goto Github PK
View Code? Open in Web Editor NEWAWS-based application for scraping the Maryland Judiciary Case Search
License: GNU General Public License v3.0
AWS-based application for scraping the Maryland Judiciary Case Search
License: GNU General Public License v3.0
Defendant personally-identifiable information (PII) fields (name, address, DOB) should be pulled out into a separate table and referenced in the various case tables using the ID key from the PII table. This will need to be done for each type of case (DSCR, DSK8, DSCIVIL, CC, ODYTRAF, etc), though the PII can all be consolidated into a single table. Then a new read-only database user should be created that can read from all tables EXCEPT the PII table, which we can give to partners that don't have a compelling need to see the PII.
Since both the spider and scraper components make requests to the MJCS server, they are limited in their concurrency so as not to overload MJCS and make it unavailable to others. By default, both components are limited to 10 concurrent connections to MJCS. These defaults are arbitrary safe values; it would be better to determine safe concurrency levels based on load testing MJCS.
For each search of MJCS, the server reports the number of seconds it took to process the request:
If we were to periodically execute the same search on MJCS and track the number of seconds the search takes, and graph that versus the running concurrency of Case Harvester as we scale it up and down, that would help us see when increasing concurrency actually starts affecting MJCS server performance.
The spider and scraper component currently record several metrics that can be used to evaluate performance. These metrics should be sent to CloudWatch so they can be graphed and monitored in a dashboard.
For each spider search item, the following metrics are recorded:
For each run of the spider, the following metrics are recorded:
For each scrape, the following metrics are recorded:
This document lists the meanings of the various codes used in DSCR event history records. The DSCR parser should be extended to parse these codes from the event history comments.
To help users of the data understand what it means, especially all the legal jargon, it would be helpful to have a glossary of all the field names from the various case types along with their descriptions. For version control purposes I think this should live in a JSON file. Alternatively or in addition to this, the glossary could also be added to the database itself using COMMENT ON COLUMN
.
I don't have the legal background for this, so help would be very much appreciated.
Case Harvester currently doesn't have any tests. It would be great to add some basic unit tests using the Pytest framework. Once some tests are in place, they can be run with make test
.
The following checklist indicates the MJCS case formats we currently have parsers for:
I'm having good results by doing a see-saw approach between first and last name, and by assuming each case has a defendant/respondent. Seems to reduce clutter in the results a lot. Any interest in pr?
Currently the spider component is only run from the command line. We need a Lambda function for the spider that can be triggered:
Each method should include parameters that set the search criteria (e.g. a specific county and time range). For example, the weekly run could search for cases over the last month, while the daily run could only search for cases within the last week.
Right now the Case Harvester spider finds new cases by iteratively searching last names, but we haven't yet looked at case number patterns so we could potentially find any gaps, missing cases the spider search algorithm hasn't found. For example, if Baltimore County district criminal cases always use case numbers in the format BC-123-XXXXX, we could exhaustively search for all case numbers in that range.
I think the best way to do this would be to create a JSON file that maps the various courts and case types to their case number patterns/ranges, which Case Harvester could read in and use to look for new cases.
The README documentation needs to be update to reflect the following changes:
The (MJCS notice page)[https://mdcourts.gov/casesearch2/notice] reports on scheduled downtimes. It would be great to have a period script run (like a Lambda function on a schedule) that monitors for changes on the notice page and emails them to someone. This will allow us to make sure Case Harvester isn't running when MJCs is unavailable.
This should be automatically created during database initialization, and credentials should be pulled from secrets.json
, similar to the master and regular users.
We need a basic web presence describing the Case Harvester project, including mission, available data, how to contribute, how to report issues or give feedback, and info for researchers and journalists who want to access our database.
Down the line we would like to add sample analyses and visualizations, and also custom search functionality.
Use AWS Simple Notification Service (SNS) to notify admin via email of failed scheduled tasks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.