bcgov / housing-data-challenge-plot-scatter

BCIC-BC Stats Data Viz Challenge - Plot and Scatter Submission

This application is built using Bootstrap, Leaflet, D3, and React.

The purpose of each component is described in the source files, but a basic familiarity with React is essential. The Quick Start guide is very good.

The deployed prototype app can be viewed at http://plotandscatter.com/bcic-data-challenge/.

NB. These instructions have not been tested on Windows.

To run the project locally

1. Clone the repository.
2. In the root project folder, run npm install.
3. Once npm finishes, run npm run compile-vendors to generate the DLL. The DLL contains dependencies that are static (e.g. react itself) and so don't need to be rebuilt every time the dev server starts. This improves local webpack performance. You don't need to build the DLL again unless you add additional static dependencies to src/vendors.js.
4. Once the DLL has been built, run npm run dev.

View the app at http://localhost:3000 in your browser. Note that this assumes that there is a production instance somewhere serving the regional vector tiles. If not, follow the instructions below under "Serving the regional vector tiles" to set up a tile server on your local machine.

To deploy the project to production

Serving the regional vector tiles

The various shape files representing the regional boundaries at different aggregation levels (e.g. census division, census area, economic region) have been converted to GeoJSON. In the steps below, tippecanoe is used to convert the GeoJSON to the MBTiles vector tile format. Compared to serving and rendering the GeoJSON directly, the tile format offers extremely good performance while retaining fine boundary details even at high zoom levels. A third-party Docker container, running concurrently with the main app, is responsible for serving the vector tiles.

Install dependencies

On a Mac:

• Homebrew
• Docker for Mac
• tippecanoe: from a Terminal, run brew install tippecanoe once Homebrew is installed

On Linux:

These instructions may vary depending on the Linux distribution. The following has been tested on Ubuntu 14.04.

• Docker for Ubuntu
• tippecanoe: follow the instructions in this GitHub issue thread

Generate the vector tiles

1. From your root project directory, cd src/data/boundaries/

2. Check the geo-json subfolder, and unzip any files found there

3. Then run the following commands from the src/data/boundaries/ folder, one at a time (each might take a few minutes):

   tippecanoe -o ./vector-tiles/municipalities.mbtiles ./geo-json/Municipalities_geo.json --no-polygon-splitting
   tippecanoe -o ./vector-tiles/econregions.mbtiles ./geo-json/EconRegions_geo.json --no-polygon-splitting
   tippecanoe -o ./vector-tiles/regdistricts.mbtiles ./geo-json/RegDistricts_geo.json --no-polygon-splitting
   tippecanoe -o ./vector-tiles/census-dissemination-areas.mbtiles ./geo-json/DisseminationAreas_clipped.json --no-polygon-splitting
   tippecanoe -o ./vector-tiles/census-dissemination-blocks.mbtiles ./geo-json/DisseminationBlocks_clipped.json --no-polygon-splitting
   tippecanoe -o ./vector-tiles/census-tracts.mbtiles ./geo-json/CensusTracts_geo.json --no-polygon-splitting
   

Serve the vector tiles

1. From the same folder as in the previous step (src/data/boundaries), run the following command: docker run -v $(pwd):/data -p 8080:80 klokantech/tileserver-gl --config vector-tiles/tileserver-gl-config.json
2. This may take a few minutes to do a one-time install of the tileserver-gl docker image
3. Once you see the message Listening at http://:::80/, the tiles are available for use in the app; you can browse the tile server at http://yourdomain.com:8080 (or, if deployed locally, http://localhost:80)
4. NB Ensure you update the URLs in the Constants.MAP_BOUNDARY_INFO variable, found in src/constants.js

Setting up the main app in a Docker container

Adapted from https://nodejs.org/en/docs/guides/nodejs-docker-webapp/.

1. Make sure Docker is installed on your computer (https://www.docker.com/products/docker)
2. In Terminal, navigate to the root of this project
3. Build the Docker image: docker build -t plotandscatter/ps-bcic-data-viz .
4. Once the Docker image is built, run it: docker run -p 3001:3000 -d plotandscatter/ps-bcic-data-viz

You should now be able to access the Docker server at http://yourdomain.com:3001 (or http://localhost:3001 if running locally).

To kill a running Docker container

1. docker ps
2. Find the container ID for the image 'plotandscatter/ps-bcic-data-viz'
3. docker kill <container-id>

To clean up old Docker containers

Adapted from http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/ and http://stackoverflow.com/questions/30604846/docker-error-no-space-left-on-device.

List all containers

docker ps -a

Then manually use docker rm containerid or docker rmi image as appropriate for stopped/dangling images.

Delete exited containers

docker rmi $(docker images -f "dangling=true" -q) (if using sudo, sudo docker rmi $(sudo docker images -f "dangling=true" -q))

Delete 'dangling' images

docker rmi $(docker images | grep "^<none>" | awk "{print $3}")

For a list of data sources, see the README.md in src/data/.

housing-data-challenge-plot-scatter's Issues

Housing-data-challenge-Plot+Scatter

---

Background

The Province of British Columbia and other agencies collect a wealth of administrative data that relates to housing, including the market for housing, as a consequence of their regulatory and administrative authorities. These include data about legal ownership (Land Titles BC), the total stock of housing (BC Assessment), and taxation information (Ministry of Finance). In addition, socio-demographic information is available, such as ownership status (owner or renter), through the Census of Canada.

The British Columbia Government seeks to develop statistics that will provide greater certainty about the state of housing in the province, including the role of foreign ownership, real estate as an investment or business strategy (rather than home ownership), and insights into the regional impact of these issues. For more information see https://github.com/bcgov/housing-data-visualization-project.

We are offering this issue as a paid opportunity through our experimental procurement model, Code-with-Us and under the following terms Code-with-Us Terms {PDF}.

User Story

Government, infrastructure planners and the wider public need to interact with an online visualization platform of B.C. housing market information (using the following types of data: population data, property assessments, property transfer tax records, home owner grant records, land title records, etc.) so that they understand and make decisions based on community growth and housing data.

Acceptance Criteria

To be paid for this opportunity meet the following criteria:

• The code is mounted in a Province of B.C. GitHub repository under an open source license.

  • Preference is for Apache 2.0.

• Should be able to use Property Transfer Tax data: https://catalogue.data.gov.bc.ca/dataset/property-transfer-tax-data.

  • These data are licensed under the B.C. Government Open Data License.

• Should be able to use Census data:

  • preference is for demographics at the Census Tract level, but the ability to aggregate to higher levels of geography (e.g. municipality) would be desirable.
  • see the 2011 Census of Population data products page
  • link to Census geography files, including boundary (shape) files
  • these data and supplementary files are licensed under the Statistics Canada Open Licence.

• Should be able to use BC Assessment data (geocoded to Census geographies; assuming availability)

• Code can be run from the instructions given (e.g. from readme file)

• Must have final delivery on March 7th, with initial milestones for code commits in GitHub on February 17th and March 3rd for feedback.

Evaluation Criteria

Your visualization will be scored on the following (60 points total):

• Function: how easy is it to navigate and interact with the visualization (15 points).

• Story: how does the visualization use narrative to make the data relevant and meaningful for the audience (15 points).

• Design: how does the visualization use best design practice (e.g. good design principles, visual cues, hierarchy) so that the meaning is clear (15 points).

• Data Analysis: how does the visualization uncover patterns and relationships in the data (15 points).

Code will be scored on the following (40 points total):

• Instructions on how to install and run the code (5 points).

• Is the code documented (literate programming; maintainability) (10 points).

• Does the code run on the Province of B.C. DevOps platform (a Docker / OpenShift environment). This is much easier if you setup up the infrastructure before you start coding (5 points).

• How transferable (extensible) is the code to analyze and visualize other government data (10 points).

• Processing efficiency / responsive (5 points)

• Runs on desktop required; bonus for tablet (5 points).

Useful links

How payment works
Original competition description

Add missing topics

TL;DR

Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

Why Topic

In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means its critical that we work to make our content as discoverable as possible; Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

What to do

Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

That's in, you're done!!!

How to use

Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

Pro Tip 🤓

• If your org is not in the list below, or the table contains errors, please create an issue here.

• While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

• Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

• If your application is live, add the production URL.

Ministry Short Codes

Short Code	Organization Name
AEST	Advanced Education, Skills & Training
AGRI	Agriculture
ALC	Agriculture Land Commission
AG	Attorney General
MCF	Children & Family Development
CITZ	Citizens' Services
DBC	Destination BC
EMBC	Emergency Management BC
EAO	Environmental Assessment Office
EDUC	Education
EMPR	Energy, Mines & Petroleum Resources
ENV	Environment & Climate Change Strategy
FIN	Finance
FLNR	Forests, Lands, Natural Resource Operations & Rural Development
HLTH	Health
FLNR	Indigenous Relations & Reconciliation
JEDC	Jobs, Economic Development & Competitiveness
LBR	Labour Policy & Legislation
LDB	BC Liquor Distribution Branch
MMHA	Mental Health & Addictions
MAH	Municipal Affairs & Housing
BCPC	Pension Corporation
PSA	Public Safety & Solicitor General & Emergency B.C.
SDPR	Social Development & Poverty Reduction
TCA	Tourism, Arts & Culture
TRAN	Transportation & Infrastructure

NOTE See an error or omission? Please create an issue here to get it remedied.

Give BCStats team Admin access

Add missing topics

TL;DR

Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

Why Topic

In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means it's critical that we work to make our content as discoverable as possible. Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

What to do

Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

That's it, you're done!!!

How to use

Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

Pro Tip 🤓

• If your org is not in the list below, or the table contains errors, please create an issue here.

• While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

• Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

• If your application is live, add the production URL.

Ministry Short Codes

Short Code	Organization Name
AEST	Advanced Education, Skills & Training
AGRI	Agriculture
ALC	Agriculture Land Commission
AG	Attorney General
MCF	Children & Family Development
CITZ	Citizens' Services
DBC	Destination BC
EMBC	Emergency Management BC
EAO	Environmental Assessment Office
EDUC	Education
EMPR	Energy, Mines & Petroleum Resources
ENV	Environment & Climate Change Strategy
FIN	Finance
FLNR	Forests, Lands, Natural Resource Operations & Rural Development
HLTH	Health
IRR	Indigenous Relations & Reconciliation
JEDC	Jobs, Economic Development & Competitiveness
LBR	Labour Policy & Legislation
LDB	BC Liquor Distribution Branch
MMHA	Mental Health & Addictions
MAH	Municipal Affairs & Housing
BCPC	Pension Corporation
PSA	Public Safety & Solicitor General & Emergency B.C.
SDPR	Social Development & Poverty Reduction
TCA	Tourism, Arts & Culture
TRAN	Transportation & Infrastructure

NOTE See an error or omission? Please create an issue here to get it remedied.