Unique about node-gtfs

[brendannee]

Thanks for your patience. I was able to find the issue.

• The error messages on import were getting hidden, I pushed an updated version of node-gtfs that solves this (you don't need to update though).

The issue was that your project has specified a different version of mongoose from the one one specificed by node-gtfs. In your package.json file it calls for mongoose 5.0.6, but node-gtfs specifics 5.0.9. This was causing two versions to get installed, and causing errors on import.

The way to fix this is to specify

"mongoose": "5.0.9",

in your package.json. Delete your node_modules folder and package-lock.json and run npm install again. You can check in node_modules/gtfs/node_modules to make sure a second version of mongoose isn't getting installed.

[brendannee]

Can you let me know the GTFS file you using?

I think perhaps it got imported twice - you could try clearing your database and reimporting the GTFS to see if that helps?

[andreasvirkus]

The GTFS can be found here: http://peatus.ee/gtfs/
And my Heroku app runs with the delete before importing, you can see a query here
(might take a while to spin up):
https://tallinn-transport-api.herokuapp.com/api/area/59.435787/24.750738

so I doubt it's an issue with importing multiple times 🤔

[brendannee]

Check your database - check the stops collection - I think you may have somehow imported more than once. If you see stop_id in your stops collection more than once, this is the case.

I tried importing your GTFS file and running the same getStops query and got no duplicates:

[{"stop_id":"1634","stop_name":"Viru"},{"stop_id":"4341","stop_name":"Viru"},{"stop_id":"1291","stop_name":"Viru"},{"stop_id":"1292","stop_name":"Viru"},{"stop_id":"1290","stop_name":"Viru"},{"stop_id":"1635","stop_name":"Viru"},{"stop_id":"1627","stop_name":"Estonia"},{"stop_id":"1630","stop_name":"Estonia"},{"stop_id":"1628","stop_name":"Estonia"},{"stop_id":"4324","stop_name":"Estonia 1"},{"stop_id":"3098","stop_name":"Kaubamaja"},{"stop_id":"1293","stop_name":"Mere puiestee"},{"stop_id":"1294","stop_name":"Mere puiestee"},{"stop_id":"1629","stop_name":"Estonia"},{"stop_id":"10378","stop_name":"Estonia 2"},{"stop_id":"4334","stop_name":"Estonia"},{"stop_id":"14969","stop_name":"Estonia"},{"stop_id":"4325","stop_name":"Estonia 3"},{"stop_id":"4326","stop_name":"Estonia 4"},{"stop_id":"1304","stop_name":"Mere puiestee"},{"stop_id":"10379","stop_name":"Estonia"},{"stop_id":"4327","stop_name":"Estonia 5"},{"stop_id":"1632","stop_name":"Kaubamaja"},{"stop_id":"4328","stop_name":"Estonia 6"},{"stop_id":"10397","stop_name":"Estonia 7"},{"stop_id":"1633","stop_name":"Kaubamaja"},{"stop_id":"1286","stop_name":"Vabaduse väljak"},{"stop_id":"3890","stop_name":"Viru keskus 1"},{"stop_id":"1324","stop_name":"Vabaduse väljak"},{"stop_id":"3610","stop_name":"Kaubamaja"},{"stop_id":"3892","stop_name":"Viru keskus 3"},{"stop_id":"1631","stop_name":"Kaubamaja"},{"stop_id":"14902","stop_name":"Kaubamaja"},{"stop_id":"3891","stop_name":"Viru keskus 2"},{"stop_id":"1303","stop_name":"Mere puiestee"},{"stop_id":"3893","stop_name":"Viru keskus 4"},{"stop_id":"3894","stop_name":"Viru keskus 5"},{"stop_id":"3895","stop_name":"Viru keskus 6"},{"stop_id":"3603","stop_name":"A. Laikmaa"},{"stop_id":"1281","stop_name":"Vabaduse väljak"},{"stop_id":"1636","stop_name":"A. Laikmaa"},{"stop_id":"1280","stop_name":"Vabaduse väljak"},{"stop_id":"1295","stop_name":"A. Laikmaa"},{"stop_id":"1282","stop_name":"Vabaduse väljak"},{"stop_id":"1289","stop_name":"Vabaduse väljak"},{"stop_id":"1297","stop_name":"A. Laikmaa"},{"stop_id":"1298","stop_name":"Hobujaama"},{"stop_id":"10402","stop_name":"A. Laikmaa"},{"stop_id":"3103","stop_name":"Kivisilla"},{"stop_id":"1301","stop_name":"Hobujaama"},{"stop_id":"1318","stop_name":"Kivisilla"},{"stop_id":"3606","stop_name":"Hobujaama"},{"stop_id":"1288","stop_name":"Vabaduse väljak"},{"stop_id":"3604","stop_name":"Hobujaama"},{"stop_id":"1300","stop_name":"Hobujaama"},{"stop_id":"1319","stop_name":"Kivisilla"},{"stop_id":"3605","stop_name":"Hobujaama"},{"stop_id":"1322","stop_name":"Vabaduse väljak"},{"stop_id":"3102","stop_name":"Kivisilla"},{"stop_id":"1305","stop_name":"Hobujaama"},{"stop_id":"3402","stop_name":"Hobujaama"},{"stop_id":"1323","stop_name":"Vabaduse väljak"}]

[andreasvirkus]

Trying to debug my code, but all of a sudden I'm now getting NaN lines imported per text file.
Does this indicate there's something up with my mongoose connection?

> node server.js

Config before import: { env: 'production',
  isDev: false,
  verbose: true,
  mongoUrl: 'mongodb://localhost:27017/gtfs',
  port: '5000',
  agencyNames: [ 'Tallinna Linnatranspordi AS', 'ELRON' ],
  agencies:
   [ { agency_key: 'tallinn',
       url: 'http://www.peatus.ee/gtfs/gtfs.zip',
       exclude: [Array] } ] }
Starting GTFS import...

Starting GTFS import for 1 fileServer listening on port 5000

tallinn: Downloading GTFS from http://www.peatus.ee/gtfs/gtfs.zip
tallinn: Download successful
tallinn: Importing GTFS from C:\Users\antc\code\tallinn-transport-api\gtfs-downloads/tallinn-gtfs.zip
tallinn: Importing - agency.txt - NaN lines imported
tallinn: Importing - calendar_dates.txt - NaN lines imported
tallinn: Importing - calendar.txt - NaN lines imported
tallinn: Skipping - fare_attributes.txt
tallinn: Skipping - fare_rules.txt
tallinn: Skipping - feed_info.txt
tallinn: Skipping - frequencies.txt
tallinn: Importing - routes.txt - NaN lines imported
tallinn: Skipping - shapes.txt
tallinn: Importing - stop_times.txt - NaN lines imported
tallinn: Importing - stops.txt - NaN lines imported
tallinn: Importing - transfers.txt - No file found
tallinn: Skipping - trips.txt
tallinn: Post Processing data

It then stays pending in the last step (post processing data). You can see the code at
https://github.com/andreasvirkus/tallinn-transport-api/

[brendannee]

It sounds like there may be some kind of uncaught error. Can you tell me the version of node.js are using and the version of mongodb you are running and if you are OS X, linux or windows?

In the last output you posted, it might be possible that the actual download or unzip of GTFS had a problem since it didn't actually import any lines.

[andreasvirkus]

I'm on v9.6.0 on Windows but can test on OSX as well tomorrow on my work rig, will post back if I get a more verbose output on the Mac.

Can I do anything else to help debug this in the meanwhile?

Edit: Tested on OSX, output is exactly the same and stays in the "Post processing data" step

[andreasvirkus]

@brendannee Is there anything you'd know to suggest for me to try or any info I could provide somehow to get to the bottom of this?
Sorry to press the issue

[brendannee]

Can you attach the full source of your app (or point me to your repo) so I can see what you are running and the exact config?

On OS X, did you still get some files importing with NaN lines imported?

[andreasvirkus]

Sure thing! I also brought it up earlier in this thread: https://github.com/andreasvirkus/tallinn-transport-api/

And yup, on OSX I also got all the non-skipped files as NaN lines imported and it hangs in the last step Post processing data

[andreasvirkus]

Ah, yes! This makes total sense, since the error occurred after upgrading to 1.5.3 in regards to #90 (comment)
Embarrassed I didn't think of it before and sorry for wasting your time on this one.
Should mongoose maybe be listed as a peerDependency instead?

[andreasvirkus]

Sorry, closed a bit preemptively. Now I'm seeing an error Error: Unable to initialize Mongo bulk insert. when gtfs' lib/import.js tries to run model.collection.initializeUnorderedBulkOp()
The full error trace:

[ tallinn-transport-api | master ] ◉  npm start

> [email protected] start C:\Users\antc\code\tallinn-transport-api
> node server.js

Config before import: { verbose: true,
  mongoUrl: 'mongodb://localhost:27017/gtfs',
  agencies:
   [ { agency_key: 'tallinn',
       url: 'http://www.peatus.ee/gtfs/gtfs.zip',
       exclude: [Array] } ] }
Starting GTFS import...

Starting GTFS import for 1 fileServer listening on port 5000

tallinn: Downloading GTFS from http://www.peatus.ee/gtfs/gtfs.zip
tallinn: Download successful
tallinn: Importing GTFS from C:\Users\antc\code\tallinn-transport-api\gtfs-downloads/tallinn-gtfs.zip
C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:240
              throw err;
              ^

Error: Unable to initialize Mongo bulk insert.
    at importLines (C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:62:15)
    at Parser.parser.on (C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:238:11)
    at Parser.emit (events.js:130:15)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node server.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\antc\AppData\Roaming\npm-cache\_logs\2018-03-18T20_21_13_745Z-debug.log

Should I open up a new issue for this? Since the original 'unique' issue seems to be something on my end

[brendannee]

Good idea - I was wondering if it should be a peerDependency but it then makes the use case for less technical users where someone wants to simply ‘npm install gtfs’ and then use it as a command line tool need to also need to install mongoose. I’ll test some things out and see if a peer dependency would be feasible.

…

On Sun, Mar 18, 2018 at 13:24 ajv ***@***.***> wrote: Sorry, closed a bit preemptively. Now I'm seeing the following error: [ tallinn-transport-api | master ] ◉ npm start > ***@***.*** start C:\Users\antc\code\tallinn-transport-api > node server.js Config before import: { verbose: true, mongoUrl: 'mongodb://localhost:27017/gtfs', agencies: [ { agency_key: 'tallinn', url: 'http://www.peatus.ee/gtfs/gtfs.zip', exclude: [Array] } ] } Starting GTFS import... Starting GTFS import for 1 fileServer listening on port 5000 tallinn: Downloading GTFS from http://www.peatus.ee/gtfs/gtfs.zip tallinn: Download successful tallinn: Importing GTFS from C:\Users\antc\code\tallinn-transport-api\gtfs-downloads/ tallinn-gtfs.zip C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:240 throw err; ^ Error: Unable to initialize Mongo bulk insert. at importLines (C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:62:15) at Parser.parser.on (C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:238:11) at Parser.emit (events.js:130:15) at endReadableNT (_stream_readable.js:1101:12) at process._tickCallback (internal/process/next_tick.js:152:19) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! ***@***.*** start: `node server.js` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the ***@***.*** start script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\antc\AppData\Roaming\npm-cache\_logs\2018-03-18T20_21_13_745Z-debug.log Should I open up a new issue for this? Since the original 'unique' issue seems to be something on my end — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAF32fCtIndK7XFBopgrTnY3ip5trpDIks5tfsKVgaJpZM4ST1Vy> .

[brendannee]

If you are seeing that error, it means you still have two versions of mongoose installed.

I think peer dependency is the way to go - I updated the library to use that and it seems to install as expected - see the latest version 1.6.0.

Delete your node_modules folder and package-lock.json file and update to the latest version and this issue should be solved.

[andreasvirkus]

Interesting, rimraffing node_modules and package-lock didn't do the trick with 1.5.3, but worked fine with 1.6.0. Also all the stops are now unique (most likely since I don't have to run skip-delete anymore, which might have caused the issue earlier : )

Thank you so much @brendannee for the constant support!