blinktaginc / node-gtfs Goto Github PK

You might want to add into the readme that if you haven't configured the git repo yet, git push heroku master will fail with "fatal: 'heroku' does not appear to be a git repository". To fix:

git remote add heroku [email protected]:appname.git

I would suggest to add the possibility to pass a list for filtering the fields of mongo query.

For example this:

exports.getAgencies = (query = {}) => {
  ...
  return Agency.find(query, '-_id').lean();
};

would become this:

exports.getAgencies = (query = {}, projection='-_id') => {
  ...
  return Agency.find(query, projection).lean();
};

The bundle imported had 2 agencies:

agency.txt

"agency_id","agency_name","agency_url","agency_timezone","agency_lang","agency_phone"
"NSWTrainLink","NSW TrainLink","http://www.nswtrainlink.info/","Australia/Sydney","en","131500"
"SydneyTrains","Sydney Trains","http://www.sydneytrains.info/","Australia/Sydney","en","131500"

config.js

{
  "mongo_url": "mongodb://127.0.0.1:27017/tfnsw",
  "agencies": [
    {
      "agency_key": "SydneyTrains",
      "path": "../gtfs-files/sydneytrains.zip"
    },
    {
      "agency_key": "NSWTrainLink",
      "path": "../gtfs-files/sydneytrains.zip"
    }
  ]
}

Entire bundle is imported twice:

$ npm run import

> [email protected] import M:\tfnsw\node-gtfs
> node lib/import.js

SydneyTrains: Importing GTFS from ../gtfs-files/sydneytrains.zip
SydneyTrains: Importing - agency.txt - 2 lines imported
SydneyTrains: Importing - calendar_dates.txt - No file found
SydneyTrains: Importing - calendar.txt - 117 lines imported
SydneyTrains: Importing - fare_attributes.txt - No file found
SydneyTrains: Importing - fare_rules.txt - No file found
SydneyTrains: Importing - feed_info.txt - No file found
SydneyTrains: Importing - frequencies.txt - No file found
SydneyTrains: Importing - routes.txt - 133 lines imported
SydneyTrains: Importing - shapes.txt - 82724 lines imported
SydneyTrains: Importing - stop_times.txt - 799299 lines imported
SydneyTrains: Importing - stops.txt - 1236 lines imported
SydneyTrains: Importing - transfers.txt - No file found
SydneyTrains: Importing - trips.txt - 42371 lines imported
SydneyTrains: Post Processing data
SydneyTrains: Completed
NSWTrainLink: Importing GTFS from ../gtfs-files/sydneytrains.zip
NSWTrainLink: Importing - agency.txt - 2 lines imported
NSWTrainLink: Importing - calendar_dates.txt - No file found
NSWTrainLink: Importing - calendar.txt - 117 lines imported
NSWTrainLink: Importing - fare_attributes.txt - No file found
NSWTrainLink: Importing - fare_rules.txt - No file found
NSWTrainLink: Importing - feed_info.txt - No file found
NSWTrainLink: Importing - frequencies.txt - No file found
NSWTrainLink: Importing - routes.txt - 133 lines imported
NSWTrainLink: Importing - shapes.txt - 82724 lines imported
NSWTrainLink: Importing - stop_times.txt - 799299 lines imported
NSWTrainLink: Importing - stops.txt - 1236 lines imported
NSWTrainLink: Importing - transfers.txt - No file found
NSWTrainLink: Importing - trips.txt - 42371 lines imported
NSWTrainLink: Post Processing data
NSWTrainLink: Completed
All agencies completed (2 total)

Rows are duplicated:

> db.agencies.count()
4
> db.calendars.count()
234
> db.routes.count()
266
> db.shapes.count()
165448
> db.stops.count()
2472
> db.stoptimes.count()
1598598
> db.trips.count()
84742

First, I am not used to code with Node.js so maybe I am missing something. However I was researching about tasks, third libraries or something and nothing appeared.

I hope you can help me. It occurs when I try to call this command-line sentence.

gtfs-import --skipDelete --configPath gtfs-config.json

And it shows

/usr/local/lib/node_modules/gtfs/lib/import.js:51
const removeDatabase = async task => {
                             ^^^^
SyntaxError: Unexpected identifier
    at Object.exports.runInThisContext (vm.js:76:16)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/gtfs/lib/gtfs.js:2:20)
    at Module._compile (module.js:570:32)

I am looking at the source code of findXXXbyDistance functions and the distance in radians is found with the following line:

var radiusInDegrees = Math.round(radius / 69 * 100000) / 100000;

Assuming Radius is the radius of the area to be searched and not the Earth's radius; Where is the number 69 coming from? I tried to come up with the number but I simply couldn't. Do you mind giving some insight please?

Thanks

Love this project, thank you for making it available!

In my opinion, one great extension would be to add support for GTFS-realtime (https://developers.google.com/transit/gtfs-realtime/). I'm happy to start developing those features myself but thought it wise to check if you already have similar functionality in progress for this or another library. I'm still fairly new to the Transit domain so perhaps there are challenges with GTFS-realtime that I'm still not aware of.

Would GTFS-realtime support be in scope for this library?

I've attempted many different combinations of /:agency/:route_id/:stop_id and all either return with

"error": "No times for agency/route/stop/direction combination."

or

TypeError: Cannot read property '0' of null
    at model.module.exports.secondsToTime (…/gogobus/node_modules/gtfs/lib/utils.js:32:34)
    at SchemaString.SchemaType.applyGetters (…/gogobus/node_modules/gtfs/node_modules/mongoose/lib/schematype.js:512:22)
    at model.Document.get (…/gogobus/node_modules/gtfs/node_modules/mongoose/lib/document.js:700:18)
    at model.Object.defineProperty.get [as departure_time] (…/gogobus/node_modules/gtfs/node_modules/mongoose/lib/document.js:1185:45)
    at …/gogobus/node_modules/gtfs/lib/gtfs.js:409:34
    at Array.forEach (native)
    at Promise.<anonymous> (…/gogobus/node_modules/gtfs/lib/gtfs.js:408:23)
    at Promise.<anonymous> (…/gogobus/node_modules/gtfs/node_modules/mongoose/node_modules/mpromise/lib/promise.js:162:8)
    at Promise.EventEmitter.emit (events.js:95:17)
    at Promise.emit (…/gogobus/node_modules/gtfs/node_modules/mongoose/node_modules/mpromise/lib/promise.js:79:38)

This includes attempts to call the api using more combinations of /:agency/:route_id/:stop_id after setting up a fresh clone of this repo and following the steps to import gtfs data via node ./scripts/download into a mongo db running locally and run the example express app via node ./examples/express/index.js. Stop times aren't working in either app, though every other api call to routes or stops by location or id works as is expected in both apps.

mongoing in shows more than enough stopTimes in each agency but alas,

No [stopTimes] for you!
-Soup Nazi

Here's a link to my public repo containing the implementation of this module if you're curious as to any potential hangups.

• https://github.com/jeffwesson/gogobus

It's me again! I stumbled across another issue that I haven't been able to fix.

Some route shape_pt_sequences are either being omitted, or are being added to a different one. I consulted with somebody on this, and they said it's most likely due to a callback not finishing before another async request begins.

This issue only happens on certain datasets, such as bay-area-rapid-transit, and not on others, such as bloomington-transit. I verified this by cross-referencing the shapes.txt for the files, and then the db.shapes.find() method in mongo.

I didn't check all the datasets that seem to be returning properly to see if all shape_pt_sequences returned correctly.

Here are my results:

Proper results

bloomington-transit:
from shapes.txt:
shape_id,shape_pt_lat,shape_pt_lon,shape_pt_sequence
1930,39.15466,-86.49328,143 (zero indexed, so proper number)

mongoDB query:
> db.shapes.find({agency_key: 'bloomington-transit', shape_id: '1930'}).count()
144

Dataset variation results

bay-area-rapid-transit:

from shapes.txt:
shape_id,shape_pt_lat,shape_pt_lon,shape_pt_sequence
01_shp,37.599782,-122.386655,2379

mongoDB query:
> db.shapes.find({agency_key: 'bay-area-rapid-transit', shape_id: '01_shp'}).count()
2370

Variation within same dataset

ac-transit:

from shapes.txt:
shape_id,shape_pt_sequence,shape_pt_lat,shape_pt_lon
Z0048,250198,37.790022,-122.393717 -> Counted manual and has 531 entries

mongoDB query:
> db.shapes.find({agency_key: 'ac-transit', shape_id: 'Z0048'}).count()
531

from shapes.txt:
shape_id,shape_pt_sequence,shape_pt_lat,shape_pt_lon
10013,680014,37.696816,-122.125840 -> Counted manually and has 730 entries

mongoDB query:
> db.shapes.find({agency_key: 'ac-transit', shape_id: '10013'}).count()
727

So, as you can see, something is up here, but I don't know what exactly. Thanks in advance!

I'm using the following code to try to get the Caltrain data. Apparently, it works but when checking the data I've noticed that something went wrong. Checking the raw stops file I see 31 stops with platform_code as NB and 31 as 'SB'. After importing and querying, I see 31 and 33, respectively.
The 2 extra stops for SB are 2 stations that don't have a value for platform_code in the raw file and somehow it's defaulting to SB.

"use strict";

const mongoose = require('mongoose');
const gtfs = require('gtfs');
const config = {
  mongoUrl: 'mongodb://url',
  agencies: [{
    agency_key: 'Caltrain',
    url: 'http://www.caltrain.com/Assets/GTFS/caltrain/CT-GTFS.zip',
    exclude: [
      'shapes'
    ]
  }]
};

module.exports = {
  updateData: updateData
};

function updateData(req, res) {
  mongoose.Promise = global.Promise;
  mongoose.connect(config.mongoUrl);
  gtfs.import(config, (err) => {
    res.json({
      result: err || "Success"
    });
  });
}

Config:

{
  "mongoUrl": "mongodb://localhost:27017/gtfs",
  "agencies": [
    {
      "agency_key": "openov",
      "url": "http://gtfs.openov.nl/gtfs/gtfs-openov-nl.zip"
    }
  ]
}

Output

gtfs-import --configPath ./gtfs-import.json

Starting GTFS import for 1 file
openov: Downloading GTFS from http://gtfs.openov.nl/gtfs/gtfs-openov-nl.zipbuffer.js:504
    throw new Error('"toString()" failed');
    ^

Error: "toString()" failed
    at Buffer.toString (buffer.js:504:11)
    at Request.<anonymous> (~/gtfs-importer/node_modules/request/request.js:1145:39)
    at emitOne (events.js:101:20)
    at Request.emit (events.js:191:7)
    at IncomingMessage.<anonymous> (~/gtfs-importer/node_modules/request/request.js:1091:12)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:188:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

When downloading and extracting first it works fine

I tried to pull routes somewhere in Melbourne within a mile radius.

eg. http://loc.iamraja.com:8000/getRoutesByDistance?lat=-37.866612&lon=144.993121&radius=1

But the script returned data including routes from other agencies too(Adelaide, Perth etc). I think this because the route query is not supplied with agency_id

Please let me know if you need any more info.

Importing a GTFS file always ends up with

ns=0xFATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
16b2fe47a1e1 <String[14]: gtfs.stoptimes>,ops=0x24d80f184...
Abort trap: 6

I also switched to a GTFS file with only 18MB, same result.

getTripsByRouteAndDirection function has an issue.

getTripsByRouteAndDirection: function(agency_key, route_id, direction_id, service_ids, cb) {
    var query = {
      agency_key: agency_key,
      route_id: route_id,
      direction_id: direction_id
    };
    // if (__.contains([0, 1], direction_id)) {
    //   query.direction_id = direction_id;
    // } else {
    //   query.direction_id = {
    //     $nin: [0, 1]
    //   };
    // }

    if (service_ids && service_ids.length) {
      query.service_id = {
        $in: service_ids
      };
    }
    Trip.find(query, cb);
  },

When I modify like above, it works.

I am trying to use the getTimesByStop method, but I am getting the following exception at the bottom of the issue.

I first had to fix the following mongoose query since the sort was invalid (it wasn't an object):
...
function findTimes(cb){
var query = {
agency_key: agency_key,
stop_id: stop_id
}
, timeFormatted = utils.timeToSeconds(today);

  StopTime
    .find(query)
    .where('trip_id').in(trip_ids)
    .asc('departure_time', 'ascending')

...
to
.sort({departure_time: -1})
...

After that I am seeing the following promise related issue. Any thoughts on this?

TypeError: undefined is not a function
at module.exports.getTimesByStop.query.agency_key (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\lib\gtfs.js:320:9)
at _asyncMap (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\async\lib\async.js:226:13)
at async.eachSeries.iterate (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\async\lib\async.js:130:21)
at _asyncMap (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\async\lib\async.js:223:17)
at async.series.results (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\async\lib\async.js:550:34)
at Promise.module.exports.getTimesByStop (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\lib\gtfs.js:407:13)
at Promise.onResolve (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\mongoose\node_modules\mpromise\lib\promise.js:162:8)
at Promise.EventEmitter.emit (events.js:95:17)
at Promise.emit (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\mongoose\node_modules\mpromise\lib\promise.js:79:38)
at Promise.fulfill (C:\dev\projects\nodejs\angularexpressapp\node_modules\gtfs\node_modules\mongoose\node_modules\mpromise\lib\promise.js:92:20)

Thanks for all the work put into this project!

I am using the express example app (LINK) that you previously supplied, and I've tried using the shapes endpoint, and it's failing to return any shapes.

For instance, the api call http://localhost:3000/api/shapes/kolumbus/1307/ returns:

["shape_ids","shapes"]

I've verified that the shapes are indeed in the database.

I'll try to look into it further, but any idea what might be causing this?

Thanks!

Is there a way to get all stops per agency_key?

I added a new method but not sure if I need it:

/*
   * Returns an array of all stops
   */
  getAllStops: function(agency_key, cb) {

    Stop.find({
      agency_key: agency_key      
    }, cb).sort({stop_name: 1});
  }

download.js ln 181:

csv().fromPath() is no longer valid in the 0.2.x branch of node-csv-parser. This would instead need to use csv().from.path()

Try to start a worker thread for 4R ?
https://github.com/bliksemlabs/rrrr

Hey guys,

I've been trying to import MTA data, and there are different GTFS files for different regions of the city that are under the same agency with the same agency_key. When I try to import these files, basically the datasets write over the previous ones.

Has anybody tried import MTA data with success?

Hello,

I'm new to mongodb and I have struggle to fetch gtfs from my mondodb database using the gtfs library. Here is what I've done so far :

Installed node-gtfs using npm and importing my gtfs data using this config file :

{
  "mongoUrl": "mongodb://localhost:27017/gtfs_swiss",
  "agencies": [
    {
      "agency_key": "gtfs_swiss",
      "url": "https://opentransportdata.swiss/dataset/b408f747-9838-4c05-bb98-10dac3996f17/resource/ec665c51-122a-47e7-9fb9-3df88cead4eb/download/gtfsfp20182018-02-28.zip"
    }
  ]
}

run the gtfs-import successfully filled my database.

Then I created a nodejs server in order to do some tests :

const express = require("express");
const app = express();
const port = process.env.PORT || 5656;

const gtfs = require('gtfs');
const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/gtfs_swiss');

// routes go here
app.get('/', (req, res) => {
    gtfs.getAgencies()
    .then(agencies => {
        console.log('success')
        // Do something with the array of `agencies`
    })
    .catch(err => {
        console.log('error')
    });
})

app.listen(port, () => {
    console.log(`http://localhost:${port}`)
})

But when I access my server with the port 5656, it goes to an infinite page loading. After some tests I figured out that the server was stuck in the getAgencies method. I don't think it's a problem between nodejs and mongodb as when I run my server using node server.js, I get an answer in the mongo output when it connects with mongoose.connect('mongodb://localhost:27017/gtfs_swiss'); .

Below is my mongod version :

db version v3.6.2
git version: 489d177dbd0f0420a8ca04d39fd78d0a2c539420
OpenSSL version: OpenSSL 1.0.1u-fips  22 Sep 2016
allocator: tcmalloc
modules: none
build environment:
    distmod: 2008plus-ssl
    distarch: x86_64
    target_arch: x86_64

I tried with others gets methods with no success. Thanks for the help in advance!

I would like to propose you to separate the gtfs lib and the example app in two different github repositories, so that there is a clear distinction between the two, and one can simply deploy the example app without the need to download the library.

The way they are packaged now is counterintuitive for me :).

I couldn't spot any function available in the package that retrieves all the stops available.

I added this to the gtfs package:

    // Copied and tweaked the agencies
    // function
    getAllStops: function(cb) {
        Stop.find({}, cb);
    },

Then tried using it as:

    var all_stops;
    router.get('/stops', function(req, res) {
        gtfs.getAllStops(function(err, data) {
            if (!err) {
                all_stops = data;
                console.log(all_stops);
            } else {
                console.log(err);
            }
        })
        res.json({ stops: all_stops });
    })

I get an empty response from the query. All the other queries worked as usual though

The trips.txt file from Chicago's Metra looks like this:

trip_id, arrival_time, departure_time, stop_id, stop_sequence, pickup_type, drop_off_type, center_boarding, south_boarding, bikes_allowed, notice
BNSF_BN1200_V1_A,  04:30:00,  04:30:00,  AURORA,  1,  0,  0,  0,  0,  1,  0

The calendar.txt file from Philadelphia's SEPTA rail division looks like this:

service_id, monday, tuesday, wednesday, thursday, friday, saturday, sunday, start_date, end_date
M1, 1, 1, 1, 1, 1, 0, 0, 20170910, 20180113

node-gtfs doesn't let me import and query either of them, because of the leading whitespace. I started special-casing those agencies and issuing special queries, but then I realized that Metra was even worse than I thought.

Would it be possible for node-gtfs to strip leading whitespace from all the field names and values?

When looking at /api/stops/:agency/:route_id endpoint and implementation getStopsByRoute(), it can be implied that the results is expected to return stops for ALL directions for the given agency and route_id when direction_id is null or not specified

This is also as suggested by @iyulaev in his comment

This doesn't really work - direction_id : null should match ALL direction IDs! (Ivan TODO)

Currently, both calls returns the same results as direction_id = 0, leaving out stops that might appear in another direction_id but not in direction_id = 0

[
   { stop_id: STOP1, ... },
   { stop_id: STOP2, ... }.
   ...
]

Can I suggest the results returning something similar to below, where it returns stops for EVERY direction_id for given agency+route_id combination?

[
    { 
        direction_id: 0, 
        stops: [ { stop_id: STOP1, ...}, {stop_id: STOP2, ... }, ... ]
    },
    { 
        direction_id: 1, 
        stops: [ { stop_id: STOP100, ...}, {stop_id: STOP99, ... }, ... ]
    },
    ....
]

What do you guys think?

Thanks

Hi there,

I'm having trouble when importing GTFS using gtfs-import. I have an error TypeError: Cannot convert undefined or null to object\transit\TransitData\gtfs-downloads/gtfs_swiss-gtfs.zip. Below is my files:

config.json:

{
  "mongoUrl": "mongodb://localhost:27017/gtfs_swiss",
  "agencies": [
    {
      "agency_key": "gtfs_swiss",
      "url": "http://transitfeeds.com/p/sbb-cff-ffs/793/latest/download",
      "exclude": [
        "fare_attributes",
        "fare_rules",
        "feed_info",
        "frequencies",
        "shapes"
      ]
    }
  ]
}

gtfs-import output:

C:\Users\Dany\Desktop\transit\TransitData>gtfs-import

Starting GTFS import for 1 file
gtfs_swiss: Downloading GTFS from http://transitfeeds.com/p/sbb-cff-ffs/793/latest/download
gtfs_swiss: Download successful
TypeError: Cannot convert undefined or null to object\transit\TransitData\gtfs-downloads/gtfs_swiss-gtfs.zip
    at Function.assign (<anonymous>)
    at executeOperation (C:\Users\Dany\AppData\Roaming\npm\node_modules\gtfs\node_modules\mongodb\lib\utils.js:386:14)
    at Collection.remove (C:\Users\Dany\AppData\Roaming\npm\node_modules\gtfs\node_modules\mongodb\lib\collection.js:1239:10)
    at NativeCollection.(anonymous function) [as remove] (C:\Users\Dany\AppData\Roaming\npm\node_modules\gtfs\node_modules\mongoose\lib\drivers\node-mongodb-native\collection.js:139:28)
    at Promise.all.models.map.model (C:\Users\Dany\AppData\Roaming\npm\node_modules\gtfs\lib\import.js:54:35)
    at Array.map (<anonymous>)
    at removeData (C:\Users\Dany\AppData\Roaming\npm\node_modules\gtfs\lib\import.js:53:29)
    at Object.module.exports [as import] (C:\Users\Dany\AppData\Roaming\npm\node_modules\gtfs\lib\import.js:324:13)
    at <anonymous>

The mongodb connection seems to be ok as gtfs-import successfully created collections in my mongodb database. See image attached

But it seems that the error occurs when gtfs-import tries to move the data. Any help would greatly be appreciate.

I was uploading this data set (url: http://www.bart.gov/dev/schedules/google_transit.zip) with version 0.4.5. It threw the followinf error when processing transfers.txt.

[Error: Number of columns on line 8 does not match header]

I checked the file and saw line 8 is an empty line at the end of the file.

I found this issue also appears in version 0.1.1

I was uploading san-francisco-municipal-transportation-agency (with version 0.1.1 and 0.1.0) and it threw

san-francisco-municipal-transportation-agency: Importing data - routes
san-francisco-municipal-transportation-agency: Importing data - shapes
[Error: Number of columns on line 104007 does not match header]

Again, line 104007 appeared to be an empty line at the end of the file shapes.txt

Hello,

first I would like to thank you for your very good work here! The import and several queries worked instantly and just as expected.
However, I noticed that the function getStoptimes() needs the stop_id as a required parameter. In my case I would like to query for all stoptimes for a specific trip. So the parameter stop_id won't let me do this easily.
Maybe I am missing something here and I am sure there was a reason for you to make stop_id required, but in my case I don't want to filter for that, but only for the trip_id instead.
Thanks!

skipDelete default value is "undefined" if not specified in config.json file, while in the readme file is specified that default value is "true".

const task = { exclude: agency.exclude, agency_key: agency.agency_key, agency_url: agency.url, path: agency.path, downloadDir: path.resolve('./downloads'), db, skipDelete: config.skipDelete, log: (message, overwrite) => { log(${overwrite !== true ? '\n' : ''}${task.agency_key}: ${message}); } };

// Override using --skipDelete command line argument or skipDelete in config.json if (task.skipDelete) { task.log('Skipping deletion of existing data'); } else { await removeDatabase(task); }

Hi,

I have a problem with gtfs.getStops() / gtfs.getStopsAsGeoJSON() methods.
I use these in this way:

 const routeIds = [......];
 gtfs.getStopsAsGeoJSON({
                agency_key: 'myAgency',
                route_id: {
                    $in: routeIds
                }
            })   

If I push ~30 items to the routeIds it is working fine (and takes ~ 3 mins) but with 40 (or more) items ... it seems that nothing happens.. (I waited more than 1 hour). I got no errors or logs..

What is the reason? What should I check? Thank you in advance!

Hi,

I was testing the code installed correct and with default agency in config
when i tried to import then following error come:
node version 6.9.2

gtfs-import --configPath ./config.json
/usr/lib/node_modules/gtfs/lib/import.js:14
const cleanupFiles = async task => {
^^^^
SyntaxError: Unexpected identifier
at Object.exports.runInThisContext (vm.js:76:16)
at Module._compile (module.js:542:28)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at Object. (/usr/lib/node_modules/gtfs/lib/gtfs.js:2:20)
at Module._compile (module.js:570:32)

I followed the tutorial and got an error in this line:

node ./scripts/download
Starting caltrain
caltrain: Download successful

/Users/myself/Dropbox/Projekte/Departures/node-gtfs/scripts/download.js:193
db.collection(GTFSFile.collection, function(e, collection){
^
TypeError: Cannot call method 'collection' of null
at /Users/myself/Dropbox/Projekte/Departures/node-gtfs/scripts/download.js:193:12

Any help is appreciated.

Hey there,

I was attempting to use this npm module to import the transit data for the various lines here in NYC, everything was working great until I came upon the Metro-North lines which have multiple agencies, leading unfortunately to them all being imported with the same agency_key.

There are a few options to solve this but I wanted to get your opinion. I'm happy to submit a PR but am not sure the best way to support multiple agencies since agency_key is used so heavily throughout the queries.

http://pastebin.com/rKHQJ0Bp

I have been trying to get this working for hours now. It just doesn't print anything to the console on lines 75 and 76.

Hi, I need to use getTripsByRouteAndDirection and getStops function in the lib/gtfs.js. When I use them, i get a empty tab. Can you fix this or explain me why I have this issue ?
(I test it with some GTFS, it works only when there is wheelchair_boarding for import stops.txt in the db ?).

I'm trying to import a relatively large dataset for Prague Public Transport System. Import throws an error (TypeError: Invalid data, chunk must be a string or buffer, not object) at Importing - stop_times.txt step. The file size is 74M and it has about 2M lines. Import hangs every time on different but large and round line number (xxx lines imported). I've seen values like 400000, 500000 670000 lines.

OS: Windows Server 2012R2
Node: 8.1.3
gtfs: 0.11.0
mongo db: v2.4.3

My config.json

{
    "agencies": [
        {
            "agency_key": "DPP",
            "exclude": [
                "fare_attributes",
                "fare_rules",
                "feed_info",
                "frequencies",
                "shapes"
            ],
            "url": "http://opendata.iprpraha.cz/DPP/JR/jrdata.zip"
        }
    ],
    "mongoUrl": "mongodb://localhost:27017/gtfs",
    "skipDelete": false,
    "verbose": true
}

Node output

D:\GRAYCROW\PROJECTS\IDOS\gtfs\dist>node index.js
Starting GTFS import for 1 file
DPP: Downloading GTFS from http://opendata.iprpraha.cz/DPP/JR/jrdata.zip
DPP: Download successful
DPP: Importing GTFS from D:\GRAYCROW\PROJECTS\IDOS\gtfs\dist\downloads/DPP-gtfs.zip
DPP: Importing - agency.txt - 17 lines imported
DPP: Importing - calendar_dates.txt - 72 lines imported
DPP: Importing - calendar.txt - 70 lines imported
DPP: Skipping - fare_attributes.txt
DPP: Skipping - fare_rules.txt
DPP: Skipping - feed_info.txt
DPP: Skipping - frequencies.txt
DPP: Importing - routes.txt - 387 lines imported
DPP: Skipping - shapes.txt
D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\node_modules\mongodb\lib\utils.js:123
    process.nextTick(function() { throw err; });
                                  ^

TypeError: Invalid data, chunk must be a string or buffer, not object
    at WriteStream.Socket.write (net.js:696:11)
    at text (D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\lib\import.js:283:70)
    at importLines (D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\lib\import.js:232:17)
    at bulk.execute.err (D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\lib\import.js:74:5)
    at handleCallback (D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\node_modules\mongodb\lib\utils.js:120:56)
    at D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\node_modules\mongodb\lib\bulk\unordered.js:462:22
    at handleCallback (D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\node_modules\mongodb\lib\utils.js:120:56)
    at resultHandler (D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\node_modules\mongodb\lib\bulk\unordered.js:408:11)
    at D:\GRAYCROW\PROJECTS\IDOS\gtfs\node_modules\gtfs\node_modules\mongodb-core\lib\wireprotocol\2_4_support.js:538:34
    at _combinedTickCallback (internal/process/next_tick.js:95:7)
    at process._tickCallback (internal/process/next_tick.js:161:9)

Mongo output

C:\mongodb\bin\mongod.exe --help for help and startup options
Tue Jul 04 15:22:44.713 [initandlisten] MongoDB starting : pid=11428 port=27017 dbpath=\data\db\ 64-bit host=endurance
Tue Jul 04 15:22:44.714 [initandlisten] db version v2.4.3
Tue Jul 04 15:22:44.714 [initandlisten] git version: fe1743177a5ea03e91e0052fb5e2cb2945f6d95f
Tue Jul 04 15:22:44.714 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1'
) BOOST_LIB_VERSION=1_49
Tue Jul 04 15:22:44.714 [initandlisten] allocator: system
Tue Jul 04 15:22:44.715 [initandlisten] options: {}
Tue Jul 04 15:22:44.720 [initandlisten] journal dir=\data\db\journal
Tue Jul 04 15:22:44.721 [initandlisten] recover : no journal files present, no recovery needed
Tue Jul 04 15:22:44.746 [initandlisten] waiting for connections on port 27017
Tue Jul 04 15:22:44.746 [websvr] admin web console waiting for connections on port 28017
Tue Jul 04 15:22:48.928 [initandlisten] connection accepted from 127.0.0.1:58287 #1 (1 connection now open)
Tue Jul 04 15:22:48.928 [initandlisten] connection accepted from 127.0.0.1:58288 #2 (2 connections now open)
Tue Jul 04 15:22:48.943 [initandlisten] connection accepted from 127.0.0.1:58289 #3 (3 connections now open)
Tue Jul 04 15:22:48.943 [initandlisten] connection accepted from 127.0.0.1:58290 #4 (4 connections now open)
Tue Jul 04 15:22:48.943 [initandlisten] connection accepted from 127.0.0.1:58291 #5 (5 connections now open)
Tue Jul 04 15:22:48.959 [initandlisten] connection accepted from 127.0.0.1:58292 #6 (6 connections now open)
Tue Jul 04 15:22:53.743 [initandlisten] connection accepted from 127.0.0.1:58294 #7 (7 connections now open)
Tue Jul 04 15:22:53.743 [initandlisten] connection accepted from 127.0.0.1:58295 #8 (8 connections now open)
Tue Jul 04 15:22:53.743 [initandlisten] connection accepted from 127.0.0.1:58296 #9 (9 connections now open)
Tue Jul 04 15:22:53.743 [initandlisten] connection accepted from 127.0.0.1:58297 #10 (10 connections now open)
Tue Jul 04 15:23:10.748 [conn7] remove gtfs.stoptimes query: { agency_key: "DPP" } ndeleted:510000 keyUpdates:0 numYields: 977 locks(micros) w:31485811 1
6995ms
Tue Jul 04 15:23:44.926 [conn8] insert gtfs.stoptimes ninserted:1 keyUpdates:0 locks(micros) w:83 157ms
Tue Jul 04 15:23:44.926 [conn7] insert gtfs.stoptimes ninserted:1 keyUpdates:0 locks(micros) w:157652 157ms
Tue Jul 04 15:24:00.325 [conn4] end connection 127.0.0.1:58290 (9 connections now open)
Tue Jul 04 15:24:05.430 [conn5] end connection 127.0.0.1:58291 (8 connections now open)
Tue Jul 04 15:24:10.609 [conn6] end connection 127.0.0.1:58292 (7 connections now open)
Tue Jul 04 15:24:15.917 [conn1] end connection 127.0.0.1:58287 (6 connections now open)
Tue Jul 04 15:24:20.927 [conn3] end connection 127.0.0.1:58289 (5 connections now open)
Tue Jul 04 15:25:45.895 [conn7] end connection 127.0.0.1:58294 (4 connections now open)
Tue Jul 04 15:25:45.898 [conn8] end connection 127.0.0.1:58295 (3 connections now open)
Tue Jul 04 15:25:45.914 [conn2] end connection 127.0.0.1:58288 (2 connections now open)
Tue Jul 04 15:25:45.914 [conn9] end connection 127.0.0.1:58296 (2 connections now open)
Tue Jul 04 15:25:45.914 [conn10] end connection 127.0.0.1:58297 (2 connections now open)

Hi,

I'm not the most familiar with Mongo nor Mongoose, but is it currently possible (or would it be possible to add) to fetch only unique (I think Mongoose has distinct for it?) results?

Currently querying

 gtfs.getStops({
    within: {
      lat: 59.435787,
      lon: 24.750738,
      radius: 0.3
    }
  }, {
    _id: 0,
    stop_name: 1,
    stop_id: 1
  })
  .then(stops => res.json(stops))

Results in a lot of duplicated results from my GTFS:

[{"stop_id":"1634","stop_name":"Viru"},{"stop_id":"1634","stop_name":"Viru"},
{"stop_id":"1634","stop_name":"Viru"},{"stop_id":"4341","stop_name":"Viru"},
{"stop_id":"4341","stop_name":"Viru"},{"stop_id":"4341","stop_name":"Viru"},
{"stop_id":"1291","stop_name":"Viru"},{"stop_id":"1291","stop_name":"Viru"},
{"stop_id":"1291","stop_name":"Viru"},{"stop_id":"1292","stop_name":"Viru"},
{"stop_id":"1292","stop_name":"Viru"},{"stop_id":"1292","stop_name":"Viru"},
{"stop_id":"1290","stop_name":"Viru"},{"stop_id":"1290","stop_name":"Viru"},
{"stop_id":"1290","stop_name":"Viru"},{"stop_id":"1635","stop_name":"Viru"},
{"stop_id":"1635","stop_name":"Viru"},{"stop_id":"1627","stop_name":"Estonia"},
{"stop_id":"1627","stop_name":"Estonia"},{"stop_id":"1627","stop_name":"Estonia"},
{"stop_id":"1630","stop_name":"Estonia"},{"stop_id":"1630","stop_name":"Estonia"},
{"stop_id":"1630","stop_name":"Estonia"},{"stop_id":"1628","stop_name":"Estonia"},
{"stop_id":"1628","stop_name":"Estonia"},{"stop_id":"1628","stop_name":"Estonia"},
...]

Hi, I'm getting a node error when I run "npm run download";

Utah Transit Authority: Importing data - stop_times.txt
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
Aborted

It looks like it has something to do with my stop_times.txt file, which is 29.7 mb.
I attached the npm-debug.log.
npm-debug.txt

Hi, I tried creating a simple example that would download my local agency's timetables and just print the available stops to console. I set mongoose.promise in my code, but still get a warning when I call gtfs.getStops(). My code is:

const config = require('./config.json');
const gtfs = require('gtfs');
var mongo = require('mongodb');
var mongoose = require('mongoose');

mongoose.Promise = global.Promise;

mongoose.connect(config.mongoUrl, {useMongoClient: true}).then(function() {
    console.log('connected')
    gtfs.getStops({
        agency_key: 'dpp'
    })
    .then(stops => {
        console.log(stops);
    });
}).catch(err => console.log(err));

config.json:

{
  "mongoUrl": "mongodb://localhost:27017/gtfs",
  "agencies": [
    {
      "agency_key": "dpp",
      "url": "http://opendata.iprpraha.cz/DPP/JR/jrdata.zip"
    }
  ]
}

I am using mongoose 4.11.8 and I ran gtfs.import and it reported no errors, and I did not find any problems in the db either. I found this issue on mongoose github and it seems there is a workaround, but it would require inserting
mongoose.promise = global.Promise
into almost every file that accesses the db, including gtfs files. Have you encountered this issue before or do you have any more feasible workaround?

Thanks

Hi,

I appreciate the work of this repository. Great stuff!

I was wondering if you ever considered refactoring this package to a plugin architecture. In particular, plugins for different sources to import the data and query from. Would be cool if others could write a mysql, firebase, postgres plugin if they don't like the default mongo.

I would be willing in contributing to help make this happen also.

Since @5d7f9b896df95bc5306c5af22d720af3bc75e5fa, the download script doesn't work anymore.

The output of the script before that commit was something like:

$ node ./scripts/download.js
trentino-trasporti-esercizio-spa: Starting
trentino-trasporti-esercizio-spa: Downloading
trentino-trasporti-esercizio-spa: Download successful
trentino-trasporti-esercizio-spa: Importing data - agency
trentino-trasporti-esercizio-spa: Importing data - calendar_dates
trentino-trasporti-esercizio-spa: Importing data - calendar
trentino-trasporti-esercizio-spa: Importing data - No fare_attributes file found
trentino-trasporti-esercizio-spa: Importing data - No fare_rules file found
trentino-trasporti-esercizio-spa: Importing data - No feed_info file found
trentino-trasporti-esercizio-spa: Importing data - No frequencies file found
trentino-trasporti-esercizio-spa: Importing data - routes
trentino-trasporti-esercizio-spa: Importing data - No shapes file found
trentino-trasporti-esercizio-spa: Importing data - stop_times
trentino-trasporti-esercizio-spa: Importing data - stops
trentino-trasporti-esercizio-spa: Importing data - transfers
trentino-trasporti-esercizio-spa: Importing data - trips
trentino-trasporti-esercizio-spa: Post Processing data
trentino-trasporti-esercizio-spa: Completed
All agencies completed (1 total)

since @5d7f9b896df95bc5306c5af22d720af3bc75e5fa, the output is just:

$ node ./scripts/download.js
trentino-trasporti-esercizio-spa: Starting
trentino-trasporti-esercizio-spa: Downloading
trentino-trasporti-esercizio-spa: Download successful
All agencies completed (1 total)

and the database isn't touched at all.

Could you please bump the version so that it is possible to use the latest patch for the download.js script?

I am seeing the following issue:

[Error: go-transit: Unzip failed 'unzip' is not recognized as an internal or ext
ernal command,
operable program or batch file.
]
All agencies completed (1 total)

I amended the error console log to include the stderr output. I am running off a windows 8 machine. Is the unzip you are invoking for the local environment or say through the file io package for node?

The method that is failing is:
function processFile(e, response, body){
if(response.statusCode == 200){
console.log(agency_key + ': Download successful');

    //remove old text files and unzip file
    var unzip = 'unzip ' + downloadDir + '/latest.zip -d ' + downloadDir;
    exec(unzip, function(e, stdout, stderr) {
      if(!e && !stderr){
        console.log(agency_key + ': Unzip successful');
        cb(null, 'download');
      } else { 
        cb(new Error(agency_key + ': Unzip failed ' + stderr), 'download');
      }
    });

  } else {
    cb(new Error('Couldn\'t download files'));
  }
}

Any suggestions?

Hey guys, I'm wondering if it's possible to skip the shapes.txt import and download all together. It always causes my server to slow down and hang and I don't really need this file for my project. Just wondering if this is possible.

Thanks.

Hi, I have some troubles with the getStopsByRoute function in the lib/gtfs.js module.
Currently it takes about 20 seconds in my deployment to execute (heroku app + mongolab db). I think this is caused by the inner function getTrips.

I cannot understand why you got in so much troubles by sampling the population just to get the list of trips for a given route. Of course I am missing something, but I would like to understand what.

Just for the sake of comparison, you can find at https://github.com/youtux/node-gtfs/commit/e26cf655e83f640e0f38ad050136135d2afa5e92 a version I came up with which make the same query last 4 seconds instead of 20.

I plan to implement versions of some functions to get GeoJSON
What do you think about it?

Hi, I really like this package however I'm noticing that this two queries run very very slow like almost a minute: getRoutesByStop and getStopsByRoute.

I have 1381 stops and 87 routes in my mongodb.

At application load I have my 'From' combo box filled with all available stops and it gets them quick enough (1.7s).
Then on 'From' blur I'm calling a function to first getRoutesByStop to receive all routes that serves the 'From' stop and then on each route I'm calling getStopsByRoute to receive final list of 'To' stops that you can get to directly from the 'From' stop. This gets (1.5 - 2 minutes)

I'm getting the data eventually but it's extremely slow.

Any suggestions on how to improve it?

I was wondering if it'd be possible to add a batch endpoint so I could cache all the information client side?

I'm having some trouble while trying ti import a GTFS to my database.... Here you are what happens:

Error: Cannot find configuration file at D:\workspace-node\node-gtfs\config.json. Use config-sample.json as a starting point, pass --configPath option

Hi guys - noticed a bug with getStoptimesByStop while testing. Running latest Chrome and testing on both local MongoDB via node but also a heroku dev server + GLITCH. Both environments fail. Query tested:

My Query:

gtfs.getStoptimesByStop ( 'myAgencyNameHere' , 'LA' , 'ETOILE' )
    .then(stoptimes => {
        console.log(stoptimes);
     }

All arguments here are made up for this example but my actual values were verified. Issue here is that nothing gets returned. No stop times and no error(s). Honestly no idea why it hangs.

Checked the source code and located the error within /lib/gtfs/stop-times.js - the first Promise does not pass the "Pending" stage.

Original:

exports.getStoptimesByStop = (agencyKey, routeId, stopId, directionId, cb) => {
  if (_.isFunction(directionId)) {
    cb = directionId;
    directionId = undefined;
  }
  return new Promise(() => {
        if (agencyKey === undefined) {
          new Error('No agencyKey specified');
        } else if (stopId === undefined) {
          new Error('No stopId specified');
        } else if (routeId === undefined) {
          new Error('No routeId specified');
        }
  })

My changes:

• Removed empty promise
• Passed back in resolve and reject
• added reject cases
• removed the cb and added a catch(err) at the bottom for err trace

exports.getStoptimesByStop = (agencyKey, routeId, stopId, directionId, cb) => {
  if (_.isFunction(directionId)) {
    cb = directionId;
    directionId = undefined;
  }
  return new Promise((resolve, reject) => {
    if (agencyKey && routeId && stopId) {
      resolve ();
    } else {
        if (agencyKey === undefined) {
          reject( new Error('No agencyKey specified') );
        } else if (stopId === undefined) {
          reject ( new Error('No stopId specified') );
        } else if (routeId === undefined) {
          reject ( new Error('No routeId specified') );
        }
      }
  })

// skipping to end of stack ...

.catch (err => {
    console.log(err);
}

And so

Also... this is my first issue report on Git so I apologize for anything incorrect - tried to be as helpful as possible. If this change ends up being useful, could I submit a PR and submit? That too would be my first!

And obviously, thanks so much for the project.

Hi all,
Currently I using the GTFS latest version but I got an issue that is GTFS require 'extract-zip' module, and the 'extract-zip' module require 'fs' module which is no longer used. So, how can I resolve this dependency issue ?