worldbank-transport / open-transit-indicators

An open-source tool to support transport agencies in planning and managing public transit systems

open-transit-indicators

An open-source tool to support transport agencies in planning and managing public transit systems

Build Status

Installation

Vagrant

If you're using Vagrant, then installation is as simple as cloning the repo and then issuing vagrant up from the root of the repository directory.

To change the default amount of memory allocated to the vagrant machine, set the environment variable OTI_VAGRANT_MEMORY to the preferred size, in MB.

Server installation

If you're installing directly onto a dedicated server, follow these steps after cloning the repo on an ubuntu machine version 12.04:

1. Run the following commands:

sudo apt-get update
sudo apt-get -y install git

2. Clone the project into the directory you want the project

cd <DIRECTORY TO DOWNLOAD/INSTALL PROJECT>
git clone -b master https://github.com/WorldBank-Transport/open-transit-indicators.git

3. Edit deployment/provision.sh and change the line saying: PROJECTS_DIR="/projects". The text /projects should be replaced with the absolute path of the location where you've cloned the repository. For example, if your repository folder is located in /home/myusername/open-transit-indicators, the line should become PROJECTS_DIR="/home/myusername". You can use pwd to determine this directory.

4. Move into the open-transit-indicators directory with the following command:

cd open-transit-indicators

5. From the repository folder (project directory), issue the command (you may need to enter your password):

• For development: sudo ./deployment/provision.sh development
• For production: sudo ./deployment/provision.sh production

AMI Generation

If you want to generate an Amazon Web Service's (AWS) Amazon Machine Image (AMI) you can use Packer to handle the provisioning.

Use the following instructions to do so:

1. Create an AWS account if you do not have one already
2. Get the API keys for a new user or an existing user in your AWS account
3. Download and install Packer
4. Navigate to the deployment/packer directory
5. Copy the open-transit-vars.json.example file and save it as open-transit-vars.json
6. Edit open-transit-vars.json with the API keys downloaded in step 2
7. Run the following command to generate a new AMI: packer build -var-file=open-transit-vars.json open-transit-indicators.json PLEASE NOTE: Running this command will cause resources to be created in AWS and will cost money
8. Once the process finishes installing (could take up to an hour), make note of the AMI ID
9. Launch the AMI using the AWS EC2 management console and browse to the public DNS hostname provided by Amazon

open-transit-indicators's Issues

Some map tiles do not load when the angular app is opened on a non-map page

Reproduce:

• Navigate directly to: /#/indicators/data
• Click the transit tab
• Some map tiles will not load and the map will be zoomed out

No console errors thrown or map tile 404's. Leaflet scope object looks ok.

Legend sometimes shows on settings page

Occasionally, when going to the 'Settings' page after a fresh provisioning, the legend shows in the corner, although the map is not in the background.

Route information in stops interactivity

On the Transit Map page, we used to have info with each stop about which routes used it. Did this go away at some point?

Transit System Length chart should be Horizontal Stacked

setup_angular npm install fails on EC2

Creates directories owned by root that cannot be accessed

If indicator job crashes, further jobs get stuck at 'queued'

One of my indicator processing jobs crashed (meaning the status got stuck at 'processing'), and when I tried to kick off another, it remained stuck in the 'queued' status forever until I deleted the existing job that was supposedly still 'processing'.

Perhaps inspecting the celery job status directly would give us more granular info about job statuses?

Set windshaft port in angular on provision

Changing the language resets map zoom

When the language dropdown is used to set the interface language, the map zooms out to show the whole world.

Scala web actors time out uploading gtfs data

Reproducable for datasets approx greater than 12mb.
wash-dc.zip, etc

Leaves app in bad state where data partially loaded, should delete partially loaded data on failure

Indicator results not drawing on map

Non-Admin Users Cannot Create Scenarios

Angular Production/Minification causes errors

When running the grunt tasks to combine and minify javascript, errors are introduced.

Geotrellis tests fail with timezone set

Geotrellis tests pass when timezone is left at default (UTC), but fail with timezone set to EST.

No authentication on Django Rest Framework API

Will need this if we decide to do more advanced tasks with the API

Geotrellis gtfs-parser should reload gtfs data from database on restart

Currently only loads data into memory during an upload

Add simulated transit system for testing

PR #243 changes values to make tests run and comments out now broken tests - it would be nice if we piggypacked off of testkit and gtfs-test fixtures to use the ad hoc transit system Rob built. It would, in addition to being easy to pop in, allow us to reason in a clearer fashion about what we should expect results to be.

TypeError with shapefile.py ErrorFactory

On encountering an error during demographic shapefile import, the error handler fails with
TypeError(u"'source_file' is an invalid keyword argument for this function",)
at the line
self.problemClass.objects.create(**params)

Length calculator is not accurate

As mentioned in 728989c, the SEPTA rail length calculation is showing a total of 551 kilometers, where it should actually be somewhere around 450.

Datetime oddities and less than optimal map for ObservedStopTimes

This one is a little weird. A potential bug was dealt with in a less than ideal way in MR #279 - it may warrant more exploration.

Basically, the less than ideal Map[Period, Map[String, Trip]] which is currently used for 'real time' trip mappings from scheduled trip IDs is used instead of the simpler Map[String, Trip] (which ought to be sufficient) because the latter returns the correct results + 24 hours. Something odd is apparently going on with dates.

Though this workaround might be hiding a more nefarious bug, progress is blocked on this merge and I've been unable to find evidence that such a bug is certain.

Average fare UI field requires an integer

The settings field for entering the average fare does not allow entering values with a decimal portion.

Real-Time data upload times out

Attempting to upload stop times for a midsize network (SEPTA bus) results in a timeout error; the celery import task takes tens of minutes.

Cryptic error if GTFS parser fails to load feed

On uploading a GTFS, if it passes validation, but the parser fails to load it, the UI returns a cryptic message that seems to indicate the feed failed validation. All feeds that pass validation should load through the parser, but we should display a more meaningful error message if it doesn't.

Scenario Created By always set to "oti-admin"

A token is passed to scala when creating a scenario. Right now, the oti-admin token is always used, which likely causes the created_by field to be set to oti-admin

SQL injection vuln in our windshaft server

There is a SQL injection vulnerabililty in Indicator.prototype.getSql of oti-indicator.js.

open-transit-indicators/js/windshaft/oti-indicator.js

Line 180 in 3f8715c

Indicator.prototype.getSql = function () {

This url successfully drops the table gtfs_stops:

http://localhost:8067/tiles/transit_indicators/0/%27%29%20as%20hi;%20drop%20table%20gtfs_stops;%20SELECT%20*%20FROM%20%28SELECT%20formatted_value%20as%20value,%20ntile%285%29%20over%20%28order%20by%20value%29%20as%20ntiles_bin,%20the_geom%20FROM%20transit_indicators_indicator%20WHERE%20type=%27/morning/route/10/160/394.png

It looks like we'll have to find a reasonable way to escape or sanitize strings. Stripping out all non alphanumerics should do it.

No system to poll for GTFS parser completion

PR #355 squashes the timeout bug seen with slightly larger systems of GTFS data and introduces a new issue: without some kind of celery polling (similar to that used in indicator calculation), the client is not going to be able to determine the status of a GTFS ingestion.

A mid-sprint card has been added to add this functionality.

Celery should delete tmp zip files after job done

Remove spatial data from indicators export

Handle zip files created on Macs

Zip files created on Macs currently include a folder called "__MACOSX" which contains Mac-OS-specific metadata. This causes the GTFS import Celery task to fail (the upload will fail with a timeout but the error is visible in the Celery logs). This folder should be ignored or deleted when uploading gtfs feeds (and other file types too, if it makes a difference).

Workaround: Run zip -d <gtfs_feed.zip> "__MACOSX/*" to manually remove the metadata from the zip file.

Scala gtfsparser should reload data from db before each indicator job

Fix Django tests

The recent authentication changes have caused all of the Django tests to break (401 errors on responses to all queries).

spray ExceptionHandler not handling exceptions

Although there is an exception handler defined in the implicit scope for the spray routes, it doesn't appear to actually catch errors. The handler translates its exception into JSON to send back, but since it's not catching exceptions, celery is getting an unexpected response, resulting in the UI displaying the error message "No JSON object could be decoded: line 1 column 0 (char 0)".

Replace route_type integer in charts with route description

DRY the transit legend color selection

Currently, these colors are defined in two places:
js/windshaft/oti-indicators.js
js/angular/app/scripts/config.js

Send CartoCSS style for indicator map with tile URL

Currently the app defines the indicator color palettes in two spots.

1. Windshaft, to draw the indicator geometries
2. Angular, to draw the map legends

Ideally, we should pass the geometry to windshaft as a get param, storing the style definitions in only one place.

App logs itself out on firefox/safari on page refresh

This is likely a trailing slash fail on one of the many endpoints we have.

'Delete Data' button in app only clears the UI

Should also send a DELETE request to somewhere to clear the database gtfs dataset

Need better handling when missing required parameters

Currently a user is still able to calculate indicators even if some required configuration parameters are missing. One particularly problematic parameter is Distance Buffer, which if not filled in will cause the Scala process to hang indefinitely.

Real-Time data file upload should not have name restriction

Attempting to upload a file with the .txt extension results in the error:

Upload Failed: {"source_file":["Uploaded filename must end in .txt_new"]}

Database Provisioning Fails on First Run

When provisioning a fresh machine setup_db.sh and the SQL files it runs against the database fails.

The problem seems to stem from the route type definitions being moved to a migration.

GTFS Parser requires agency_id

agency_id is an optional field in the agency.txt and routes.txt file, but the GTFS parser throws an exception unless the field is present

Settings check boxes - strange behavior

When loading the page, first the Demographic check box is activated, followed by Configuration. GTFS, boundary, and real-time aren't activated until I click on them. Clicking on users makes the real-time and GTFS check boxes disappear.

Cannot pip install requirements.txt

Deployment failed to install requirements for me. Two issues: first with ipdb. Though version 0.8 is specified, and pip install ipdb will install version 0.8, attempting to install with the version specified fails. The second issue is with transitfeed. It looks like the most recent version in pypi right now is 1.2.11 (not the latest, 1.2.12, which is specified in requirements.txt).

Real-time data should be optional

The indicator calculation process currently fails when the instance doesn't have real-time data uploaded.

Refactor Add/Delete city modal to use IndicatorJob rather than Indicator

Failed indicator calculation jobs stuck in 'processing' status indefinitely

If the scala indicator calculations throw an error, the celery job continues to poll indefinitely for status, and the UI shows the calculations as 'processing', and do not allow the user to cancel or retry the job.

The spray endpoint should return an error response if the job fails, allowing the celery job to update the status appropriately and end the task.

Map views have broken bounds if app is loaded on a non-map view

E.g. Directly navigate via url bar to /settings/overview then switch to /transit.
Voila! Map bounds broken. No console errors.

Check bar chart scale for charts where indicator values are less than 1

It appears that the y axis scale on bar charts defaults to [0,1] if the max indicator value displayed in the chart is less than 1.

Leaflet Layer/Overlay control has non-standard styling

Some not-so-specific classes in our main.scss override the input styles which errantly get applied to these elements.

Use the indicator data color ramp on the indicator map page geometries

Add Separate Config File for Windshaft

The config file for the Angular app /js/angular/app/scripts/config.js has hardcoded the port for the Windshaft tiler to 8067 -- this should be moved to a separate file for easier provisioning.