guin- / ickly

A search interface for NYC Restaurant Inspection Results data

Ickly

https://ickly.herokuapp.com/

Description

Ickly is a user friendly search interface into NYC restaurant health inspection data provided by NYC Open Data. Users can look up restaurants by name and get a list of detailed inspection data including grades, overall scores, and violation descriptions. It was built with Django, Django Rest Framework, React, Redux, ReactStrap, and Bootstrap.

Local Setup

Setup

Create a python3 virtual environment. Then clone the project and follow the steps below to get set up locally. The csv file found at the following path backend/DOHMH_NYC_Restaurant_Inspection_Results.csv is included so you can run migrations to see how the app behaves locally. However, it is not the most recent or same file as used in production which is much larger.

1. Install python packages

   $ pip install -r requirements.txt

2. Install frontend dependencies

   $ npm install

3. Create a secret_settings file to store personal database settings to be used in backend/ickly/settings

4. Set up your local postgres database and run migrations

Development

1. From the backend directory start backend server at localhost:8000

   $ ./manage.py runserver

2. From the root directory run npm start

   $ npm start

Testing

1. Run frontend tests

   $ npm test

2. Run back end tests

   $ cd backend $ pytest

Features

• Users can search for restaurants by name
• Users can click on a restaurant to get information on its inspections and violations.
• Search form has typeahead functionality for easier search
• Restaurant addresses are included in drop down results so users can find a specific restaurant if it is part of a chain, or if multiple restaurants have the same name.
• When a user selects a restaurant, the page will autoscroll to the inspections section as well as the currently open inspection to ensure violations are in view when a user expands the inspection for more information.
• The typeahead is automatically cleared and mobile keyboard is autoclosed when the typeahead becomes unfocused.

Architecture

Frontend Architecture

• React for UI components
• Redux using react-redux for frontend state management
• react-router for frontend routes
• Reactstrap and native CSS for styling

Backend Architecture

• Django project served by gunicorn
• Minimal template for serving initial page load
• Django Rest Framework API -- See API documentation here for list of endpoints and models
• PostgreSQL Database

Deployment

• Deployed on Heroku on heroku-16 stack using python and node buildbacks

Problems Encountered & Solutions

I wanted users to be able to search for a restaurant by name and see all of its inspections data. The dataset from NYC Open Data was a CSV file whose rows corresponded to inspections and separate violations. However, these rows did include a 'camis' field which was a unique identifier for a business. I wanted to transform this data to match the data models I wanted for Businesses and Inspections and I needed to get all of the unique businesses.

To do so I wrote a custom Django data migration using a generator expression to get the data in the schema I wanted in a PostgreSQL database. I wrote up a blog post on this including the custom migration file. I then created a small DRF backend API with a search filter for my frontend to consume.

ickly's Issues

Contacts page - prod

In production, the images on the contacts page is broken

Also...should maybe have just a little more content

data migration to populate Business and Inspection tables

• download a full data dump of inspection data and include it in your repo
• use makemigrations to generate an empty data migration
• edit the data migration to add logic which will load the inspection data, run through it row by row, and populate the business and inspection tables
• run the migration locally and then check the content of the business and inspection tables to make sure they were populated correctly. The number of records in the inspection table should match the number of rows in your original data dump.

webpack loaders

add loaders for sass, css, images, fonts etc to webpack config

research and fix loaders to use both url-loader and file-loader
http://survivejs.com/webpack/loading-assets/loading-images/

redux tests

Might not be worth it to test this. Test tries to make actual call upon calling the action creator and throws error because it expects to fetch an absolute url. Lots of ugly workarounds, and don't want to include anything that changes/adds to the actual code for the sake of getting a test working.

https://medium.com/@kellyrmilligan/testing-async-actions-in-redux-with-isomorphic-fetch-and-fetch-mock-35f98c6c2ee7

https://medium.com/@ferrannp/unit-testing-with-jest-redux-async-actions-fetch-9054ca28cdcd

nock/nock#431

https://medium.com/@colinlmcdonald/relative-url-testing-react-redux-with-nock-1a1ac31afabf

duplicate dates in inspection list

https://stackoverflow.com/questions/43741936/merge-objects-in-same-array-based-on-common-value-in-javascript

https://stackoverflow.com/questions/44918989/javascript-merge-array-of-objects-resulting-object-values-in-array

https://stackoverflow.com/questions/35903850/combine-json-arrays-by-key-javascript

Update bootstrap typeahead library

• update bootstrap typeahead library
• add new required isLoading prop for AsyncTypeahead component

Search Form Styling

• Desktop/Tablet: Medium width, centered
• Mobile: Full width, centered
• Typeahead options list match width of form
• Ensure address text doesn't overflow off screen

Design ideas:
If bootstrap styles can be easily overridden, possibly transparent, single underline?
Think about contrast with background/accessibility

display address under business name in search result options

options choices in the dropdown display the business name in bold with the address underneath on a separate line

• use renderMenuItemChildren prop

schema migration for businesses table

• write a Business model with necessary attributes and data types
• use makemigrations tool to generate a schema migration for this model
• run the migration to generate the table locally, and check to make sure the table schema looks correct

/v1/businesses endpoint

The goal of this ticket is to get a working version of the v1/businesses endpoint that you can demo through the DRF explorer

• define structure of business object
• urls.py entry
• BusinessViewSet: pulls data from DOHMH and returns a list of business dictionaries
• BusinessSerializer (if you need it)

Clean Landing Page Background

Full Screen section

• Background image behind Header Text and Search Form
• Use a srcset/ look into picture tag for mobile/desktop image sizes
• Maybe first just start with a full screen section with background color

Typeahead select works with keyboard enter

When selecting a business the select should work with on enter on keyboard. It currently only works with mouse click.

inspection list view

InspectionsList Component:

Renders:

List of Inspection components
Table that allows for expanding rows
https://allenfang.github.io/react-bootstrap-table/example.html#expand
Table headers with inspection attributes
initial row:
inspection date, score, grade
expanded row:
vertically stacked list
inspection type, action, violation code, violation description, critical, grade date

Inspection
inspection date, inspection type, action, violation code, violation description, critical, score, grade, grade date

• BusinessSearchContainer
  • SearchForm
  • Business
  • InspectionsList

To Do:

• find proper place to import css for the bootstraptable
• Toggle visibility of BootstrapTable on a business selection
• InspectionsList component renders BootstrapTable component with inspections prop as value of data prop
• expandable rows display only data for that row
• find a solution for empty data fields within the BST
• InspectionsList component tests

webpack dev server and react hot reloading

frontend routes to about and contact

• configure react-router
• create routes to /about and /contact
• about dummy component
• contact dummy component
• route back to / works
• test boilerplate for about and contact components
• fix behavior of navbar on mobile to collapse hamburger on select
• going back to / triggers page reload

** BusinessSearch test is not working due to TypeAhead using requestAnimationFrame which is not available in jsdom
see below:
ericgio/react-bootstrap-typeahead#185

FE tests for business selection

Add front end tests for business selection feature

Test react components:
https://semaphoreci.com/community/tutorials/getting-started-with-tdd-in-react
http://redux.js.org/docs/recipes/WritingTests.html
http://ericnish.io/blog/how-to-unit-test-react-redux-components/

• test BusinessSearch renders SearchForm and Business components
• test SearchForm component props
• test Business component

Test redux (redux tests in another ticket)

Redo `/v1/businesses` endpoint

• make a new buinesses endpoint based on the new Business model
• support list and detail views
• set up ModelViewSet
• set up ModelSerializer
• implement a Filter on your viewset which will search by name
• add index on Business.name field

error alert uses reactstrap

error alert in BusinessSearch renderErrorMessage function uses reactstrap component

set up DRF

• add rest_frameworkto installed apps
• add api root api/v1/ to root url config
• DRF permissions?
• create a businesses/ business list endpoint

landing page

Create a super basic landing page

• create a root App component that will render all other components
• render it in index.js

Landing Page Renders:

• React-bootstrap components:
  • nav*
  • header (this will be changed to something prettier)
  • a search form: use https://github.com/ericgio/react-bootstrap-typeahead?

*Implement nav functionality for an about page with react-router-bootstrap in another ticket?
http://stackoverflow.com/questions/35687353/react-bootstrap-link-item-in-a-navitem

nav using reactstrap

Functional nav using reactstrap and react router

Design:
Mobile:
Hamburger
Solid dark Background, light text.
Desktop:
transparent background, light text
Brand left, Nav: About, Contact pulled right

business endpoints are slow

Business list and detail endpoints are taking many seconds to respond, when ideally they both should be responding in less than a second. Need to figure out what is causing the slowdown and fix it, but the likely issue is an inefficient database lookup.

About and Contact components

drf search filter

search filter to search by dba as a query parameter

is this possible with a queryset that is not a django model?

in theory this should allow react typeahead component to hit an endpoint like this when onSearch fires instead of initially searching the entire list
http://localhost:8000/api/v1/businesses?search={query} or
http://localhost:8000/api/v1/businesses?dba={query}

SearchForm component fetches and displays data from endpoint

User types in a business name.

• The SearchForm component calls the businesses api endpoint and returns a list of results that match the users search.
• Each result shows business name.
• The user can select a business. Selected business and its data is available for business detail /violation list view.

redux stores business selection

• create BusinessSearch container component
• move logic from SearchForm into BusinessSearch component (state gets passed down as props)
• fetchBusiness action creator dispatches actions and makes the api call.
• rootReducer returns app state
• create store in app.js
• mapStateToProps --> BusinessSearch container has selected business available as a prop
• create Business component (just create one to make sure you can access the data as
  a prop) Actual business detail component/view will be done in another ticket.
• add PropTypes to components
• clean up code

Entry Point:

index.js
provider and create store here

Action Creators:

actions/index.js

requestBusinessDetail —> fetches business detail endpoint for selected business
receiveBusinessDetail —> receives business detail data

redux-thunk
actionCreator can return a function --> this function is executed through the middleware. Here we can make api calls using fetch.

• check if this is necessary: use babel-polyfill and fetch from isomorphic-fetch for better browser support

fetchBusiness(business)
dispatches requestBusinessDetail
fetch call here
dispatches receiveBusinessDetail
handles success and failure states

Design the State Shape:

{
     businessDetail: 
          isFetching: False
          item: { business object }
          lastUpdated: date
}

Reducers:

reducers/index.js
import actions

businessDetail: returns default empty state or requestBusinessDetail, receiveBusinessDetail, business object

const rootReducer = combineReducers({
businessDetail
})

Presentational React Components:

SearchForm
BusinessDetail

• gets selected business object passed down as props

Container Components:

BusinessSearch

• move handleSearch, renderMenuItemChildren , and handleTypeaheadResultClick logic here
• pass these down as props to SearchForm component

Component Hierarchy:

• BusinessSearchContainer
  • SearchForm
  • BusinessDetail
    • InspectionsList

if you have props as an object you can pass it a spread operator in jsx to pass the whole props object {...props}. Its easy to accidentally pass props that are not needed this way.

auto-scroll to inspection on expand

When an inspection is expanded, we should scroll the view so that it is at the top of the screen, or at least as high as it can go. This is nice because it will keep the user from needed to scroll up manually. Especially useful in mobile view.

bug - back navigation from contacts

Steps to reproduce:

1. Click on the contacts page
2. Click through to an external page from one of the contact images (i.e. github)
3. Click back on the browser
4. The server tries to navigate to /contacts which sends an error

update readme

business detail view and inspections list view design

After a user selects a business from the dropdown typeahead search form, the business detail view appears below the search form.

Component Hierarchy:

• BusinessSearchContainer
  • SearchForm
  • Business
  • InspectionsList
    • Inspection

Business Component:
renders:
Heading: Business name in bold
address, phone, cuisine description below in smaller font

InspectionsList Component:
renders:
List of Inspection components
Table that allows for expanding rows
https://allenfang.github.io/react-bootstrap-table/example.html#expand
initial row:
inspection date, critical, score, grade date, grade
expanded row:
inspection type, action, violation code, violation description

Panel with a table
Panel heading: Inspections
Table fills the panel.
Table headers with inspection attributes
Table has striped and bordered props

Inspection
inspection date, inspection type, action, violation code, violation description, critical, score, grade, grade date

schema migration for Inspections table

• create model for Inspections, with correct attributes and data types (including M:1 connection to the Business model)
• run makemigrations to generate a schema migration for this model
• run the schema migration locally and check the table schema to make sure it looks right

/v1/businesses integration tests

Write a simple integration test for this endpoint using pytest, Django test server and vcr.py:

Build two sets of tests, one for the list response and one for the detail response:

• test response status code (i.e. should be 200)
• test response body (i.e. should have correct number of objects, objects have correct attributes, etc)

nav styling

Design:
Mobile:
Hamburger
Desktop:
transparent background, light text
Brand left, Nav: About, Contact pulled right

https://stackoverflow.com/questions/43381596/bootstrap-4-navbar-color

set up front end test framework

A little quick research on testing React:
Mocha, chai, + enzyme
Jest?

• run npm update for react and react-dom to keep peer dependencies happy

• get an example test working with mocha/chai
• get an example test working using enzyme and a react component
• update readme

potentially useful links:
https://semaphoreci.com/community/tutorials/testing-react-components-with-enzyme-and-mocha

Header/Nav Wrapper based on route

https://www.sitepoint.com/community/t/how-to-change-the-header-style-depending-on-page-in-reactjs/284676/3

All routes:
Header:
Nav

'/' Route:
Header

• HeroHeader
  • Intro text
  • BusinessSearch container
    • Search Form
      Business
      Inspections

'/about/' Route:

• create Hero component
  • viewport height background hero image
  • Ickly header text
  • Search Form
• create a Header component
  • contains Nav
  • renders Hero component if route is / ie:

business detail view

Business Component:
renders:

• Business name heading
• address, phone, cuisine description below in smaller font
• use styled-components for custom css
• add some padding to business component div
• phone number display formatting
  https://github.com/yahoo/react-intl
  https://github.com/s-yadav/react-number-format

inspection list view styling

• text does not overflow when size of list group items changes in expanded row on different viewports

update packages, fix vulnerabilities

Update to bootstrap 4 and reactstrap

get vcr working

Seems there were some problems with the VCR setup so we should get that up and running

favicon

django project skeleton

• create django project
• create database
• configure database settings
• create requirements.txt

business detail view styling

FE: Error handling on Business selection promise chain

In the Promise chain to handle business selection, we should add some error handling which will flash a Bootstrap error message to the user on 4xx or 5xx server response.

• error is added to state and displays in ui
• make error alert dismissible
• reset state after alert is dismissed
• find nicer way to reset state --> separate logic into two reducers?

Business Inspection edge endpoint

• InspectionSerializer
• Implement BaseEdgeViewSet to create edge endpoints
  • parent_field variable
  • returns query set of all items on the parent
• Implement BusinessInspectionEdgeEndpoint
  • returns all Inspection objects associated with given parent (business)
• test coverage for endpoint

deployment issues/notes

manage.py not in the same directory as Procfile
https://stackoverflow.com/questions/16416172/how-can-i-modify-procfile-to-run-gunicorn-process-in-a-non-standard-folder-on-he
https://stackoverflow.com/questions/20400388/django-twoscoops-project-skeleton-on-heroku-via-gunicorn-how-to-set-procfile?rq=1

webpack.prod.config
https://webpack.js.org/guides/production/

set two buildpacks:
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app

http://barkas.com/2016/set-environment-variables-activating-virtualenv/

https://devcenter.heroku.com/articles/nodejs-support
heroku-postbuild - heroku runs npm install then run prod build
https://devcenter.heroku.com/articles/django-assets
collectstatic should automatically run during the build

TODO:

• fix staticfiles for contact images
• fix staticfiles for fonts
• console warning for line that allows hot reloading
• upgrade database
• upgrade dyno
• update README

background image

update redux store with inspections data

Redux store has inspections data

• BusinessSearch container fetches business inspection edge endpoint for selected business:
  api/v1/businesses/<parent_id>/inspections/
• updates redux store with list of inspections
• actions for request, success, and failure states
• action types
• write separate reducer for this
• use combineReducers() to refactor main reducer

Make Inspections data available to component

• map inspections array state to props

Redux state shape:

{
business:
  isFetching: bool,
  selectedBusiness: { },
  error: { },
inspections: 
  error: { },
  inspections: [{ }, { }, ... ]
}

** figure out #38 to write tests for redux async actions

sort inspections list by date

custom sort function? https://allenfang.github.io/react-bootstrap-table/example.html#sort
https://stackoverflow.com/questions/30691066/sort-a-string-date-array

https://stackoverflow.com/questions/26930692/lo-dash-sortby-array-of-dates-in-string-format

set up frontend

• create package.json
• basic webpack config
• install and configure django-webpack-loader
• index.html template at root url
• HelloWorld React Component
• bundle renders hello world react app
• example react-bootstrap component
• hot reloading of react components