Comments (15)
Thanks for your patience. I was able to find the issue.
- The error messages on import were getting hidden, I pushed an updated version of node-gtfs that solves this (you don't need to update though).
The issue was that your project has specified a different version of mongoose from the one one specificed by node-gtfs. In your package.json
file it calls for mongoose 5.0.6, but node-gtfs specifics 5.0.9. This was causing two versions to get installed, and causing errors on import.
The way to fix this is to specify
"mongoose": "5.0.9",
in your package.json. Delete your node_modules
folder and package-lock.json
and run npm install
again. You can check in node_modules/gtfs/node_modules to make sure a second version of mongoose isn't getting installed.
from node-gtfs.
Can you let me know the GTFS file you using?
I think perhaps it got imported twice - you could try clearing your database and reimporting the GTFS to see if that helps?
from node-gtfs.
The GTFS can be found here: http://peatus.ee/gtfs/
And my Heroku app runs with the delete before importing, you can see a query here
(might take a while to spin up):
https://tallinn-transport-api.herokuapp.com/api/area/59.435787/24.750738
so I doubt it's an issue with importing multiple times đ¤
from node-gtfs.
Check your database - check the stops collection - I think you may have somehow imported more than once. If you see stop_id in your stops collection more than once, this is the case.
I tried importing your GTFS file and running the same getStops query and got no duplicates:
[{"stop_id":"1634","stop_name":"Viru"},{"stop_id":"4341","stop_name":"Viru"},{"stop_id":"1291","stop_name":"Viru"},{"stop_id":"1292","stop_name":"Viru"},{"stop_id":"1290","stop_name":"Viru"},{"stop_id":"1635","stop_name":"Viru"},{"stop_id":"1627","stop_name":"Estonia"},{"stop_id":"1630","stop_name":"Estonia"},{"stop_id":"1628","stop_name":"Estonia"},{"stop_id":"4324","stop_name":"Estonia 1"},{"stop_id":"3098","stop_name":"Kaubamaja"},{"stop_id":"1293","stop_name":"Mere puiestee"},{"stop_id":"1294","stop_name":"Mere puiestee"},{"stop_id":"1629","stop_name":"Estonia"},{"stop_id":"10378","stop_name":"Estonia 2"},{"stop_id":"4334","stop_name":"Estonia"},{"stop_id":"14969","stop_name":"Estonia"},{"stop_id":"4325","stop_name":"Estonia 3"},{"stop_id":"4326","stop_name":"Estonia 4"},{"stop_id":"1304","stop_name":"Mere puiestee"},{"stop_id":"10379","stop_name":"Estonia"},{"stop_id":"4327","stop_name":"Estonia 5"},{"stop_id":"1632","stop_name":"Kaubamaja"},{"stop_id":"4328","stop_name":"Estonia 6"},{"stop_id":"10397","stop_name":"Estonia 7"},{"stop_id":"1633","stop_name":"Kaubamaja"},{"stop_id":"1286","stop_name":"Vabaduse väljak"},{"stop_id":"3890","stop_name":"Viru keskus 1"},{"stop_id":"1324","stop_name":"Vabaduse väljak"},{"stop_id":"3610","stop_name":"Kaubamaja"},{"stop_id":"3892","stop_name":"Viru keskus 3"},{"stop_id":"1631","stop_name":"Kaubamaja"},{"stop_id":"14902","stop_name":"Kaubamaja"},{"stop_id":"3891","stop_name":"Viru keskus 2"},{"stop_id":"1303","stop_name":"Mere puiestee"},{"stop_id":"3893","stop_name":"Viru keskus 4"},{"stop_id":"3894","stop_name":"Viru keskus 5"},{"stop_id":"3895","stop_name":"Viru keskus 6"},{"stop_id":"3603","stop_name":"A. Laikmaa"},{"stop_id":"1281","stop_name":"Vabaduse väljak"},{"stop_id":"1636","stop_name":"A. Laikmaa"},{"stop_id":"1280","stop_name":"Vabaduse väljak"},{"stop_id":"1295","stop_name":"A. Laikmaa"},{"stop_id":"1282","stop_name":"Vabaduse väljak"},{"stop_id":"1289","stop_name":"Vabaduse väljak"},{"stop_id":"1297","stop_name":"A. Laikmaa"},{"stop_id":"1298","stop_name":"Hobujaama"},{"stop_id":"10402","stop_name":"A. Laikmaa"},{"stop_id":"3103","stop_name":"Kivisilla"},{"stop_id":"1301","stop_name":"Hobujaama"},{"stop_id":"1318","stop_name":"Kivisilla"},{"stop_id":"3606","stop_name":"Hobujaama"},{"stop_id":"1288","stop_name":"Vabaduse väljak"},{"stop_id":"3604","stop_name":"Hobujaama"},{"stop_id":"1300","stop_name":"Hobujaama"},{"stop_id":"1319","stop_name":"Kivisilla"},{"stop_id":"3605","stop_name":"Hobujaama"},{"stop_id":"1322","stop_name":"Vabaduse väljak"},{"stop_id":"3102","stop_name":"Kivisilla"},{"stop_id":"1305","stop_name":"Hobujaama"},{"stop_id":"3402","stop_name":"Hobujaama"},{"stop_id":"1323","stop_name":"Vabaduse väljak"}]
from node-gtfs.
Trying to debug my code, but all of a sudden I'm now getting NaN lines imported
per text file.
Does this indicate there's something up with my mongoose connection?
> node server.js
Config before import: { env: 'production',
isDev: false,
verbose: true,
mongoUrl: 'mongodb://localhost:27017/gtfs',
port: '5000',
agencyNames: [ 'Tallinna Linnatranspordi AS', 'ELRON' ],
agencies:
[ { agency_key: 'tallinn',
url: 'http://www.peatus.ee/gtfs/gtfs.zip',
exclude: [Array] } ] }
Starting GTFS import...
Starting GTFS import for 1 fileServer listening on port 5000
tallinn: Downloading GTFS from http://www.peatus.ee/gtfs/gtfs.zip
tallinn: Download successful
tallinn: Importing GTFS from C:\Users\antc\code\tallinn-transport-api\gtfs-downloads/tallinn-gtfs.zip
tallinn: Importing - agency.txt - NaN lines imported
tallinn: Importing - calendar_dates.txt - NaN lines imported
tallinn: Importing - calendar.txt - NaN lines imported
tallinn: Skipping - fare_attributes.txt
tallinn: Skipping - fare_rules.txt
tallinn: Skipping - feed_info.txt
tallinn: Skipping - frequencies.txt
tallinn: Importing - routes.txt - NaN lines imported
tallinn: Skipping - shapes.txt
tallinn: Importing - stop_times.txt - NaN lines imported
tallinn: Importing - stops.txt - NaN lines imported
tallinn: Importing - transfers.txt - No file found
tallinn: Skipping - trips.txt
tallinn: Post Processing data
It then stays pending in the last step (post processing data). You can see the code at
https://github.com/andreasvirkus/tallinn-transport-api/
from node-gtfs.
It sounds like there may be some kind of uncaught error. Can you tell me the version of node.js are using and the version of mongodb you are running and if you are OS X, linux or windows?
In the last output you posted, it might be possible that the actual download or unzip of GTFS had a problem since it didn't actually import any lines.
from node-gtfs.
I'm on v9.6.0
on Windows but can test on OSX as well tomorrow on my work rig, will post back if I get a more verbose output on the Mac.
Can I do anything else to help debug this in the meanwhile?
Edit: Tested on OSX, output is exactly the same and stays in the "Post processing data" step
from node-gtfs.
@brendannee Is there anything you'd know to suggest for me to try or any info I could provide somehow to get to the bottom of this?
Sorry to press the issue
from node-gtfs.
Can you attach the full source of your app (or point me to your repo) so I can see what you are running and the exact config?
On OS X, did you still get some files importing with NaN lines imported
?
from node-gtfs.
Sure thing! I also brought it up earlier in this thread: https://github.com/andreasvirkus/tallinn-transport-api/
And yup, on OSX I also got all the non-skipped files as NaN lines imported
and it hangs in the last step Post processing data
from node-gtfs.
Ah, yes! This makes total sense, since the error occurred after upgrading to 1.5.3 in regards to #90 (comment)
Embarrassed I didn't think of it before and sorry for wasting your time on this one.
Should mongoose maybe be listed as a peerDependency
instead?
from node-gtfs.
Sorry, closed a bit preemptively. Now I'm seeing an error Error: Unable to initialize Mongo bulk insert.
when gtfs' lib/import.js
tries to run model.collection.initializeUnorderedBulkOp()
The full error trace:
[ tallinn-transport-api | master ] â npm start
> [email protected] start C:\Users\antc\code\tallinn-transport-api
> node server.js
Config before import: { verbose: true,
mongoUrl: 'mongodb://localhost:27017/gtfs',
agencies:
[ { agency_key: 'tallinn',
url: 'http://www.peatus.ee/gtfs/gtfs.zip',
exclude: [Array] } ] }
Starting GTFS import...
Starting GTFS import for 1 fileServer listening on port 5000
tallinn: Downloading GTFS from http://www.peatus.ee/gtfs/gtfs.zip
tallinn: Download successful
tallinn: Importing GTFS from C:\Users\antc\code\tallinn-transport-api\gtfs-downloads/tallinn-gtfs.zip
C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:240
throw err;
^
Error: Unable to initialize Mongo bulk insert.
at importLines (C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:62:15)
at Parser.parser.on (C:\Users\antc\code\tallinn-transport-api\node_modules\gtfs\lib\import.js:238:11)
at Parser.emit (events.js:130:15)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node server.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\antc\AppData\Roaming\npm-cache\_logs\2018-03-18T20_21_13_745Z-debug.log
Should I open up a new issue for this? Since the original 'unique' issue seems to be something on my end
from node-gtfs.
from node-gtfs.
If you are seeing that error, it means you still have two versions of mongoose installed.
I think peer dependency is the way to go - I updated the library to use that and it seems to install as expected - see the latest version 1.6.0.
Delete your node_modules
folder and package-lock.json
file and update to the latest version and this issue should be solved.
from node-gtfs.
Interesting, rimraffing node_modules
and package-lock didn't do the trick with 1.5.3
, but worked fine with 1.6.0
. Also all the stops are now unique (most likely since I don't have to run skip-delete
anymore, which might have caused the issue earlier : )
Thank you so much @brendannee for the constant support!
from node-gtfs.
Related Issues (20)
- Can I use the command line for query operations? HOT 6
- GTFS-R TripDescriptor HOT 5
- Working with Docker Database locked HOT 4
- Dropping/Clearing database HOT 1
- Deployment on errors HOT 5
- Invalid default csv parser option "relax" in import script. HOT 2
- Performance improvements HOT 4
- Possible performance improvement: DuckDB HOT 4
- Changing internal maxInsertVariables has significant impact on total import time HOT 5
- Agency_id defined in agency.txt but not in routes.txt results in invalid GTFS export HOT 1
- Edge deployment and SQLite HOT 4
- Not running with Deno HOT 9
- occupancyStatus from vehicle_positions HOT 6
- Persistent trip_updates when running getStopTimeUpdates HOT 13
- Deleting the db to avoid id collisions HOT 3
- Propagate RT delays to missing stop_sequences HOT 6
- updateGtfsRealtime doesn't work if one URL is down HOT 2
- Disable clean stale GTFS-Realtime data by default HOT 12
- Zip File Containing Multiple GTFS Static Zip Files HOT 3
- import.js throws an error because attribution_id is missing but the spec says it's optional HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. đđđ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â¤ď¸ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-gtfs.