transitmatters / new-train-tracker

Tracking the MBTA's new Orange, Red, and Green trains

New Train Tracker

Developed by TransitMatters Labs

Shows new MBTA Orange, Red, and Green Line trains as they come into service.

Install & Run

Dependencies:

• node 20.x (with npm 10.x)
• python 3.12
• poetry
  • Ensure poetry is using the correct Python version by running poetry env use <your python3.12 binary>

Run:

• $ npm install
• $ npm start
• visit http://localhost:5000/

To use an API key, put it in server/secrets.py or as an environment variable MBTA_V3_API_KEY

Linting

To lint frontend and backend code, run $ npm run lint in the root directory

To lint just frontend code, run $ npm run lint-frontend

To lint just backend code, run $ npm run lint-backend

Server Deployment

Additional requirements:

• nginx

Nginx serves as a reverse-proxy for the app running on localhost:5001. The Flask app is run under gunicorn and controlled by systemd, which will restart after failure or reboot automatically.

AWS Deployment

1. Make sure AWS CLI is set up and working — i.e. aws cloudformation describe-stacks | wc -l should work
2. Make sure these environment variables are set up in your shell (ask a Labs member for values if needed):

• TM_NTT_CERT_ARN (for production)
• TM_LABS_WILDCARD_CERT_ARN (for beta)

3. A key named transitmatters-ntt needs to be available in your AWS account and copied to ~/.ssh/transitmatters-ntt.pem.
4. You will also need to create an ec2-secrets.py file in the devops/ directory.
5. Run cd devops && ./deploy.sh (add -p for production) to deploy.
6. You're all set! Visit:

• https://ntt-beta.labs.transitmatters.org for beta
• https://traintracker.transitmatters.org for production

Other Deployments

This project generally fits the "Flask app" mold. Contact us if you need help: [email protected]

Support TransitMatters

If you've found this app helpful or interesting, please consider donating to TransitMatters to help support our mission to provide data-driven advocacy for a more reliable, sustainable, and equitable transit system in Metropolitan Boston.

new-train-tracker's Issues

Update Python to 3.12

Because we're running on EC2 we should be able to update to Python 3.11 or 3.12 today

Every other product at TM is using Python 3.10 now, we should at least get NTT in-line. Soon we'll be on 3.11 and 3.12

Create Ansible playbook specifically for fresh deployments

The current deploy-on-lightsail.yml Ansible playbook can only be used to update existing deployments of the Train Tracker. It wouldn't be a bad idea to have a separate playbook designed for performing fresh deployments of the app.

Extend Green Line visualization/tracking to account for GLX

GLX is currently planned to open in December for the Union Branch and May for the Medford Branch. We need to extend the GL code to take into account these new stations. @idreyn If you have any details as to what exactly needs to be done feel free to drop them here for whoever ends up picking this up.

Do something about CloudFront so we can eliminate public IPv4

If we replace CloudFront with...something else... (ALB?), we can eliminate NTT's public IPv4 address.

Ideally we would like to stop paying for a public ipv4 address dedicated to the EC2 instance

fix silverline logging

The changes to tally the new vehicles seen don't account for the logging of the silverline that's happening.
Separating the processing from the tallying would be ideal.

Add timestamps to log files

The cron task (cd /home/ubuntu/new-train-tracker && /home/ubuntu/.local/bin/pipenv run python3 -m server.history.update_history_db >> /home/ubuntu/new-train-database-updates.log) and the mbta_api call (new-train-tracker/server/mbta_api.py: print(f"Writing problematic API response to {log_path}")) both generate log files that don't have timestamps. It would be great to add timestamps to these for diagnosing issues.

Add analytics

We use Google Analytics for the data dashboard, so let's hook that up here too.

Build and attach healthcheck

NTT would ideally have a healthcheck (like the data dashboard) that UptimeRobot could hit. This could check things such as

• Do we have a successful connection to MBTA
• Has the history table updated recently?

The dashboard's equivalent is here

Make the `apple-touch-icon` work for iOS

We have an apple-touch-icon.png file in the repo which is supposed to be used when the NTT is added to the Home Screen. This...isn't working, and I can't figure out why. There's probably something I'm not doing correctly with the image itself or the link tag in template.html.

Build in GitHub Actions

Setup TravisCI auto-build/test/deploy for the site.

Add ability to filter by sub-line (GL B/C/D/E & RL Ashmont/Braintree)

This would be very helpful for when there's several trains bunched up in one rendered location, and you have to keep selecting the location to scroll through the trains, and it doesn't always scroll through all of them.

Alternatively/simultaneously, it would be nice to color code the sub-lines and expand a bunch of overlapping trains when one is selected.

Add Datadog monitoring

Just like Gobble, we should add Datadog monitoring and tracing

Database Overhaul

Should include list of npm commands and things to look out for if setting up a new instance of the project, maybe some troubleshooting tips...

• At the very least modify the README to list a running Postgres server as a dependency
• Also add some links to helpful, tested OS-specific instructions on how to set one up
• For our own purposes try to install this from scratch on an EC2/Lightsail Ubuntu instance and write down or script the steps we needed to take
• Factor any OS-agnostic steps we missed into create_history_db.py
• (Optional) Replace or supplement secrets.py with environment variables?
• #84

Update Node to 20.x

We should update to the latest node release of node 20.x

Show how busy each train is

The v3 api provides info about how busy the trains are. On the new orange and red line trains the data is given per car. We should show that on the map for the curious folks

Display average travel time between stations

It could be cool to be able to click on two stations and see the average travel time between them, especially if you're trying to schedule when you should leave to catch one of the new trains

Add labels for GL branch stations

Mobile viewports are too small to fit full station names but we could:

• Make them teeny tiny
• Abbreviate them
• Make them appear when the station dot is tapped

Clicking on a train that's on a branch line should zoom in on the branch

Right now, clicking on a train on a branch displays all of the branch stations, but they render under the map, making the display cluttered. We should instead zoom into the branch

Add expected arrival time and last update date to train modal

A lot of my friends who are train enthusiasts use Stefan on MBTA for tracking trains . They mentioned that the only things missing from making NTT an alternative to this tool are update date and expected arrival time, which the other tool provides.

Updated postcss-loader broke the world

After merging #155

ValidationError: Invalid options object. PostCSS Loader has been initialized using an options object that does not match the API schema.", " -

Actions can't clone Git repository

No mystery here, see actions/checkout#760

I'll add the workaround used by others for now

Add history and statistics page

Similar to what is being done on newtrains.today and more also
@mathcolo If you have any details as to what exactly needs to be done feel free to drop them here for whoever ends up picking this up.

(Urgent) Add codepaths to allow running the tracker without a database?

Make it easier to deploy local changes to Beta

Currently, deploy-playbook.yml is hardcoded to deploy whatever is on the org main branch - this is good for CD, but we should extend the playbook and deploy.sh to make it easier to deploy local changes without having to manually edit the playbook

Better API key management

[ ] Specify API key an with environment variable
[ ] Via the file secrets.py
[ ] Make the API calls not fail if an API key isn't present via env var or secrets.py

Save state on refresh

NTT page refreshes should send you back to the same line/mode you were already viewing. No need for URL state though, I don't think...

RL last seen not updating properly

I wonder if this is due to Ashmont shuttling? We received this to labs@.

Hi,
Earlier today (Friday, 20-October-2023) the redline new trains tracker said there were two new trains on the line. Now, when I look, (after 5:30 PM same day), it says there are no new trains, and that a new red line train was last seen 6 days ago.

That looks like a bug....

Thanks

OL strip map line becomes translucent on NEW -> ALL transition

To reproduce:

• Load the tracker
• Move to OL
• Switch from NEW to ALL trains

The strip map line will go translucent. This is a visual cue we use for the non-selected branch of GL and RL and should never happen on the OL.

Screen.Recording.2022-09-21.at.9.58.20.PM.mov

Rename `master` branch to `main`

Similar to how we've done this for other TM projects

Southbound Red line trains show destination as Ashmont/Braintree

Even once they're south of the fork, the trains list destination as Ashmont/Braintree instead of either/or.

secrets.py should not be overwritten point blank

@z0glen made a delightful change to permit NTT to run without an adjacent PostgreSQL server. That included a secrets.py modification, which the CD job overwrote on merge with the instance copy. I added Zach's change to said copy, but ideally the key would be patched in (or read from env? or elsewhere?)

Backup last_seen.json

The NTT working directory now contains last_seen.json for the "last seen" times. We should back that up automatically, in case something happens to the server.

New Train Tracker Header should not resize with page height

The new train tracker header should not resize as the page height varies (see behavior below)

Add codepaths to allow running the tracker without a database

Since we're considering using the Data Dashboard infrastructure for the history data, we can eventually wind down this repo's dependency on a locally-hosted Postgres database. As a first step, let's make our Postgres dependency opt-in — so that the app will build and run without a database available, and provide dummy/fallback data in its absence.

Add link or embed TransitMatters explanation of delay on new trainsets

Right now no trains are showing on New Train Tracker for the Orange and Red Lines. We could embed a link to an article or even just add some text explaining why that is the case.

Adding timestamp to logging- one more spot

new-train-tracker/server/mbta_api.py

Line 36 in fa5dcf9

"API returned",

Location modal for upper most stations shows up beneath sticky header on desktop

The modal that shows a train at Wonderland appears partially behind the sticky header at the top by default. I think this would be the same for other lines as well if a train is stopped at the uppermost station on the page.

Update lint action to use Node 16 and `npm ci`

To match recent changes made to the NTT

Add the Mattapan line

A user wrote in to point out that the ancient PCC cars are being retrofitted with new components, which may be of interest to the same folks who are interested in hunting down new trains. New or not, this will be a good addition as fleet delivery wears on and the NTT becomes just the "train tracker" with Blue, Silver, etc.

We could possibly tack this onto the bottom of the Ashmont strip map, though that's technically harder (and maybe not as appropriate for the use case) than just putting it into its own tab.

Add footer with dynamic version tags

The NTT currently lacks a footer - we should add one similar to the DD with links and a dynamic version tag

Eliminate Postgres

We can calculate new-trip totals in https://github.com/transitmatters/data-ingestion/, so we shouldn't need the full postgres database in this app.
We don't want to lose last-seen times though, so perhaps another solution is in order. (memory? file? s3?)