stephen-gates / gtfs

Public transport data in GTFS format with schemas, a data package and tests

GTFS

Public transportation schedules and associated geographic information for South-East Queensland.

The data is a snapshot and not planned to be kept up-to-date. The main purpose of this repository is to develop a data package and schemas for this dataset.

Data

This data is the General transit feed specification (GTFS) — South East Queensland data published by Transport and Main Roads, Queensland Government, licensed under Creative Commons Attribution sourced on 07 September 2016.

The data follows the GTFS specification and some of its extensions that define a common format for public transportation schedules and associated geographic information. The specification allows some files to be optional. It also allows some columns in the files to be optional. This means that the datapackage.json file and schemas may not work for other GTFS files.

The data is made up of a number of files.

Each data file is defined by a schema. The schemas follow the json table schema specification.

These schemas will be combined into a datapackage.json file to fully describe the data collection. The datapackage.json file will follow the data package specification.

Preparation

The data was downloaded, unzipped, and then uploaded to GitHub.

Two data files (shapes.txt and trips.txt) were too large to load into GitHub. They were truncated and uploaded. They will be adequate to use for testing valid data.

Tests

The focus of the tests is to ensure the schemas are correct. There are already GTFS data validation tools to test the data in more powerful ways than json table schemas allow.

The tests are invalid data that is used to ensure the schema detects all errors (e.g. incorrect types and violated constraints).

Results

The results can be verified using links to Good Tables. Tests include:

• testing the valid data without a schema
• testing the valid data with a schema
• testing the invalid data with a schema

Good Tables doesn't check all types of errors (yet). Somethings not checked include:

• Foreign keys. (See Good Tables #17, #8)
• Some constraints (See Good Tables#55)

Automatic Testing

The scripts and .travis.yml file are used to automatically test the data that is defined in datapackage.json. Whenever there is a change to this repository, it triggers Travis to validate the data.

The last automatic test returned

Schemas

The schemas were created using Data Packagist. Using Data Packagist:

• add some basic information about the data file (name, description, license, etc.)
• upload the data file

Data Packagist will create a datapackage.json file for you. Download this file.

Good Tables can only use a json table schema for validation (see goodtables-web #65). You can extract the json table schema from the datapackage.json file. It's this bit {fields: [...]}. Save this a separate file.

Edit the schema file with a text editor (e.g. ATOM, jsoneditoronline.org) and add constraints, refine types and formats, etc. You may like to use the json table schema schema to improve your editing experience.

Some constraints use regular expressions to define a pattern. Use a online tool to help create and test a regular expresion e.g. regexr.com or regex101.

View the Data Package

Data packages are about providing machine-readable metadata for your data. You can view a human-readable version of the data package data, and readme files using the Data Package Viewer. There are a couple of issues with the viewer including providing an incorrect link to the metadata data.okfn.org-new #9.

License

All items in this repository, apart from the data, are licensed under Creative Commons Attribution 4.0.

gtfs's Issues

Automate update of data

Instead of just having a snapshot of the data, automate the download of data from data.qld.gov.au

Split test data into smaller sets

GoodTables only returns a limited number of errors so the test data will need to be split into smaller sets.

Consider naming standard for test types:

• type and format
• constraints
  • required
  • unique
  • minimum, maximum
  • minLength, maxLength
  • pattern
  • emun
    -missing values
• primary and foreign keys, duplicates
  -structural errors
  • Undeclared header: if you do not specify in a machine readable way whether or not your CSV has a header row
  • Ragged rows: if every row in the file doesn't have the same number of columns
  • Blank rows: if there are any blank rows
  • Stray/Unclosed quote: if there are any unclosed quotes in the file
  • Whitespace: if there is any whitespace between commas and double quotes around fields
  • Empty column name: if all the columns don't have a name
  • Duplicate column name: if all the column names aren't unique

explore if it is possible to do continuous integration with CKAN

This blog explains how to do continuous integration with Github or S3 as the data source.

Explore if it is possible to do continuous integration with CKAN as the data source. Determine how to detect a data update on CKAN to trigger the data validation.

Shapes.txt is too big to show as a raw file

truncate it.

Can route_type in routes.txt be null?

test for values required in one or both columns in routes.txt

GTFS spec says file must contain at least one of route_short_name or route_long_name.
There is no obvious test for this in the json table schema.
Required:false is set on both columns

Unable to describe meaning of codes

See topic in Frictionless data forum.

Can't fix until json table schema specification supports this concept.

Add foreign keys to datapackage.json

perform continuous data integration

Automate tests to run every time the data or schemas change.
See:

• http://frictionlessdata.io/guides/validating-data/#continuous-data-integration
• http://okfnlabs.org/blog/2016/05/17/automated-data-validation.html
• http://okfnlabs.org/blog/2016/07/13/continuous-processing-with-data-packages.html

Add primary keys to schemas

To do:

• agency
• calendar
• calendar_dates
• feed_info
• routes
• shapes
• stops
• trips