amamenko / nypd-arrest-map

Data visualization application for year-to-date NYPD arrests

NYPD Arrest Map

Data visualization and analysis tool for year-to-date NYPD arrests

View Demo · Watch Video Demo · Report Issue

Introduction

The New York Police Department (NYPD) publishes data that reflect every arrest that takes place in New York City (NYC) during the current year. This dataset is manually extracted and updated on a quarterly basis by the Office of Management Analysis and Planning on NYC Open Data's NYPD Arrest Data (Year to Date) website. On this website, the NYPD states that "this data can be used by the public to explore the nature of police enforcement activity."

Every arrest that is published includes an approximate location and date, as well as demographical information about the arrestee. Other details about the nature of the arrest, such as whether a felony, misdemeanor, or violation has taken place, are also included.

Please note that occasionally, arrests in the NYPD datasets will be erroneously attributed to one borough, while having the longitude and latitude of another. These have been corrected in the NYPD Arrest Map in favor of the arrest's geocoordinates.

About the Map

The NYPD Arrest Map is an application built with the React library (using Redux for state management) as well as the Express framework for Node.js. The majority of the functionality of the website can be broken down as follows:

Server-side

• Scrapes the NYC Open Data's NYPD Arrest Data (Year to Date) page daily using Puppeteer and node-cron.
• Compares the last updated date/data numbers to those stored in Contentful.
• Parses and formats the dataset if any updates are detected.
• Modifies the Contentful data stores (if updates detected) via Contentful's Content Management API.
• Gzip-compresses and uploads the JSON dataset to a Google Cloud Storage (GCS) bucket.
• Upon request, uses Oboe.js to stream the latest JSON data from GCS to the client in chunks by means of the WebSocket API.

Client-side

• Compiles received dataset chunks and performs analyses/formatting in background threads using Web Workers.
• Serves map assets and geomarkers using deck.gl and Mapbox.
• Mounts carousel (only upon request, due to memory allocation issues) showing Google Charts of various natures (pie charts, bar charts, line charts) based on current filtered data.
• When prompted, filters currently shown arrest geomarkers and graph data in background thread.

Deployment

Server-side and client-side initially deployed to Heroku, now deployed on Render.com.

Local Development

To set up this project locally, you can follow the steps below.

Prerequisites

You will need to have the following software installed:

• npm
• Git
• Node.js

Installation

1. Get a free Mapbox API token at https://www.mapbox.com.
2. Create a new Google Cloud Storage project in the Cloud Console.
3. Create a new service account key JSON file.
4. Clone the Github repository.

   git clone https://github.com/amamenko/nypd-arrest-map.git

5. Install all server-side NPM packages.

   npm install

6. Install all client-side NPM packages.

   cd Client
   npm install

7. Webpack will throw a Conflict: Multiple assets emit to the same filename warning due to web workers creating multiple background threads and not emitting to dynamic filenames. Edit the webpack.config.js file, specifically the output key - concerning the build folder - of the returned object with the following values:

    filename: isEnvProduction
          ? 'static/js/[name].[contenthash:8].js'
          : isEnvDevelopment && '[name].bundle.js'

    chunkFilename: isEnvProduction
          ? 'static/js/[name].[contenthash:8].chunk.js'
          : isEnvDevelopment && 'static/js/[name].chunk.js'

8. Enter your Mapbox API token as a client-side environment variable.

   REACT_APP_MAPBOX_TOKEN=YOUR MAPBOX TOKEN
   REACT_APP_CONTENTFUL_SPACE_ID=YOUR CONTENTFUL SPACE ID
   REACT_APP_CONTENTFUL_ACCESS_TOKEN=YOUR CONTENTFUL ACCESS TOKEN

9. Enter your Google Cloud Storage JSON and Contentful information as server-side environment variables.

   PROJECT_ID=YOUR PROJECT ID
   PRIVATE_KEY_ID=YOUR PRIVATE KEY ID
   PRIVATE_KEY=YOUR PRIVATE KEY
   CLIENT_EMAIL=YOUR CLIENT EMAIL
   CLIENT_ID=YOUR CLIENT ID
   AUTH_URI=YOUR AUTH URI
   TOKEN_URI=YOUR TOKEN URI
   PROVIDER_CERT_URL=YOUR PROVIDER CERT URL
   CLIENT_CERT_URL=YOUR CLIENT CERT URL
   CONTENTFUL_SPACE_ID=YOUR CONTENTFUL SPACE ID
   CONTENTFUL_ENTRY_ID=YOUR CONTENTFUL ENTRY ID
   CONTENTFUL_ACCESS_TOKEN=YOUR CONTENTFUL ACCESS TOKEN
   CONTENTFUL_MANAGEMENT_TOKEN=YOUR CONTENTFUL MANAGEMENT TOKEN

10. Build for production.

    npm run build

Known Issue

iPhone 11 phones running iOS 14.1 have been found to experience "zoom breathing" when geomarkers are selected. The scatterplot layer will scale and zoom in for a moment before resetting itself. I have opened an issue with deck.gl regarding this and, although the bug is still open, the issue is otherwise fixed in iOS 14.2.

Contributing

Contributions are welcome!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/MyFeature)
3. Commit your Changes (git commit -m 'Add my feature')
4. Push to the Branch (git push origin feature/MyFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE.txt for more information.

Contact

Avraham (Avi) Mamenko - [email protected]

Project Link: https://github.com/amamenko/nypd-arrest-map

Acknowledgements

• NYPD / NYC Open Data
• deck.gl
• Mapbox
• Render.com
• React Icons
• React Alice Carousel
• React Google Charts
• Tippy.js for React
• React Radio Buttons
• React Ladda
• React Burger Menu
• ReactJS Popup
• React Circular Progressbar
• React Spinners
• react-div-100vh
• React Modal
• worker-loader
• Oboe.js
• Puppeteer
• node-cron
• csvtojson
• Google Cloud Storage
• Day.js
• express-sslify
• Best-README-Template