• Ground as altitude causes weird sort
• For items that could have the same value, need to apply a secondary sort to determine correct order

Hi @fredclausen,

Just FYI, I've found that there's a problem with the x86_64 version of the image when built by GitHub.

acarsdec and vdlm2dec both segfault when started.

I think that this is due to compiler optimisations being too aggressive, and I'm experimenting with reducing them.

I've created the branch troubleshoot_segfault for the troubleshooting works.

-Mike

Cool Stuff

• Move all of the stats page to graphs
• Map options
  • Toggle ACARS only targets
  • Show aircraft callsign on live map, with ability to toggle on/off
  • Show extended data block for target
• Side bar
  • Show planes
  • Sort by column
• Smartly position live map tool tips so they don't get cut off by edge of map
• Show decoder serial in acars/vdlm_decoder log messages
• Smooth zoom
• More zoom steps
• Dark map tile set
• Icons from tar1090
• ADSB now populates quicker on page refresh if page was active on the refresh, or clicked within 5 seconds of page load
• Legend for map colors
• Live map page now updates everything when a new message comes in, so if you have a window up showing a plane's messages it'll show up there.

Nerd stuff

• jquery all the things
• Remove all typescript expect errors (sort of...)
• Don't load live map if ADSB disabled
• Convert msg_received to object
• Save svg icon to airplane object
• stat page image test assets
• buildtest no longer minifies the code
• Handle reconnect events more smarter
• SVG to own file

Bug Fixes:

• Tooltips on live map msg box missing, search page, and alert page missing
• ACARS Hub Logo as SVG
• ADSB disabled fix nginx url
• ADSB refresh gets magically faster on web client reconnect
• Search won't fail if you include a dash in a search field, among other characters that count as a 'word boundary'
• Message total count on live map page is wrong
• Message matching (re bug above) is buggered, has been for a long time...including live messages page.
• Live Map view port size is wonky

Airline link

Welcome to Version 2.0 :)

Rather than a super-long discord message I thought it would be easier to just put this on GitHub. I've been hard at work on version 2 and it is finally time to let it out of the nest for testing. I expect, considering the huge amount of code that has been changed on the website that this will be a little buggy and require some time to get back to the stable standard that we've enjoyed with 1.x. If you find you have too many issues there are no breaking changes that will preclude you from jumping back to :latest.

The main goal for v2.0 has been to make the website more cohesive, usable on mobile, within the limits of my abilities improve the appearance, and implement "Project Eagle". More on that below.

New features that everyone will care about

• Site is now single page. No loading as you switch tabs.
• Mobile should now be a much better experience.
• Page settings / interactions are now in a pop up box, accessed via the "Page Settings" link on the banner.
• Tool tips are now actually good and not ugly! (I am way too excited about this)
• If a message matches your alert terms you will get a notification on screen
• Message tabs now dynamically size themselves to your screen width

New features, but for nerds

• Javascript has been moved over to typescript
• New build setup, including pre-commit hooks to enforce language norms and using npm
• Moved base image from debian-slim to debian-sid, which means we have Python 3.9 and newer packages all around
• Updates all 3rd party code libraries (CSS/JS) to more recent versions. Before final release I will go through again and make sure we're all up to date.
• There is now a "testing" build target and a "production" build target. Right now the only difference is that the ACARS Hub web JS and CSS is "minified", but before final release I will move all of the code in to a single JS file to improve site loading time.
• 404s are now redirected to a good page
• Backend support for Project Eagle

Requesting Feedback

• The search and stats page are in an incomplete status. Basically I moved what was in the side bar in to the main content area, but didn't really think much past that. What is everyone's thoughts on how to display that information?
• I've tested this on a Mac, iPad (12 whatever inch), iPhone 12 Pro Max for hardware, and Safari/Chrome/Firefox on the same. I'd love feedback on Windows, Android devices, and phones with smaller/bigger screens than I have access to.

KNOWN ISSUES

• If you access the site and use a "bad" URL the server redirects you (woo no more 404s :) ) the "page setting" link for Live Messages doesn't show up. As a work-around click live-messages link to fix (Build 1019)
• Live Map page cannot be directly accessed via url/adsb (Build 1019)
• There are lots of little "pixel misalignments" with some of the page elements so things might look....weird. If you see them, please report them.
• System status is going to show unhealthy. Visual bug, everything under the hood is fine. Probably. (Build 1019)
• Live Map messages may get a bit squished if there is decoded text (Build 1024)
• You may see JSON errors in your docker logs. Investigating (Build 1020)
• Browser back button doesn't take you back to previous pages (Build 1024)
• ADSB may partially display. Javascript console will show errors. Investigating. (Build 1021)
• If your ADSB_URL is malformed nginx won't start up and ACARS Hub web interface won't be accessible. (Build 1023)

TODOS

• Implement nginx (Build 1021)
• gzip server responses w/ nginx with the goal of reducing load time (Build 1021)
• Single JS file production build target for ACARS Hub web code (Build 1019)
• Finish TS optimizations (Build 1019)
• Update readme.md to include Project Eagle stuff
• Update the help page to include Project Eagle stuff and remove references to the old site layout (Build 1020)
• Move alert database search from main table to the special alert table (Build 1023)
• Smart message box sizing (resizable manually?) on live map (Build 1020)
• Chattiness/alerts in the live message marker (Deferring until v2.1)
• De Bounce alerts if page is active (Build 1019)
• Improve mouse hover information for ADSB targets (Build 1020)
• Move to using aircraft.json from external source (Build 1020)
• ADSB status (from fetch errors) in system status (Build 1021)

Project Eagle

Now that you've made it this far....lets talk about Project Eagle. Why that name...dunno, just liked it and went with it :)

Project Eagle is the start of adding in ADSB and message paring with ADSB targets. This is NOT meant to be a tar1090 replacement, and as such may feel a little...bare, compared to tar1090. There will be more stuff added to it, but keep in mind the goal here.

To enable this feature

      - ENABLE_ADSB=true
      - ADSB_URL=http://tar1090/data/aircraft.json
      - ADSB_LAT=0
      - ADSB_LON=0

Default values for each variable are

• ENABLE_ADSB: False
• ADSB_URL: URL where aircraft.json is located.

Please make sure you set the appropriate IP address and port for ADSB_URL. Most users should be fine with setting ENABLE_ADSB ADSB_LAT ADSB_LON provided you are running ACARS Hub on the same machine as you are running tar1090. If not, set ADSB_URL to the aircraft.json file from your tar1090 and ACARS Hub takes care of the rest.

You can hover over a plane and get some basic information such as the callsign. If a plane has ACARS Messages the plane is shown with a circle around it, and if you click on the target you'll get a popup that shows those messages.

ADSB Source

A valid source of readsb aircraft.json ADSB data is required. I recommend using Mike Nye's tar1090 to make it easy, although other methods of providing that are possible.

In order to make ACARS Hub ADSB matching with ACARS Messages the best it can be, your aircraft.json source needs to include tail numbers for planes, which it doesn't by default. If you do see a registration number in the callsign of an ADSB target right now it's because that is the "callsign" the plane is using. If the plane is operating under a company callsign or anything else, the registration will not be included by default in the ADSB data. To change this, please ensure your aircraft.json source has this information.

To configure Mike's tar1090 to provide this extra information, please make sure you are using an up-to-date version of his container (anything after 17 June 21 should do), and add the following to your tar1090 env variables:

      - TAR1090_ENABLE_AC_DB=true

There is a downside to enabling this feature, though, you need to keep in mind. tar1090 will use more RAM, so on RAM constrained systems you will not want to turn this on.

Where are we going from here

The main focus for v2.0 is to take what has been done and "de-bug" it. I am more than happy to log down feature requests (and I imagine there is going to be a boat load of them for Project Eagle) but it's super important to take what we've got here and make it stable before moving on.

Change Log

v2.0 Build 1018

11 June 21

• All the stuff above

v2.0 Build 1019

13 June 21

• All pages in TS/JS are now objects
• Page settings link in banner now works on page load
• ACARS Hub is now minified into a single JS file, reducing the number of times the server has to send a file, and reducing the file size of the javascript by 50%
• Can now refresh/load site from the Live Map page
• System status will now show status correctly, minus ADSB status

v2.0 Build 1020

15 June 21

• Unread alert counter now shows correctly on page load
• Moved to requiring aircraft.json from readsb. Use Mike Nye's tar1090
• Changed ADSB configuration options. See the config above.
• Mouse hover on Live Map now shows more things
• Live Map message box can be repositioned
• Live Map message box resizes itself based on screen size
• Live Map message box now remembers where you put it on screen after moved
• Live Map message box doesn't grey out background

v2.0 Build 1021

Internal Build

v2.0 Build 1022

17 June 21

• ADSB_URL formatting changed again. Sorry. Last time. If you are running tar1090 on the same machine as ACARS Hub you can remove ADSB_URL from your docker-compose file entirely. If you are not running tar1090 on the same machine as you are running ACARS Hub on you need to format the URL like a normal http URL in your browser, and it should be the full path to aircraft.json. http://192.168.31.28/data/aircraft.json for instance.
• nginx is now installed in the container. What that means is server responses should be faster because all of the files are compressed before being sent from the server.
• nginx now proxies aircraft.json requests from the browser to tar1090
• No more partial ADSB displaying
• Better ADSB pairing with ACARS Messages based on tail numbers. Please see ADSB Source above.
• ADSB status now reflected in the system status.

v2.0 Build 1023

17 June 21

• Alert page message population should now use a more efficient method for getting text matches
• Database alert table now prunes after 120 days
• Server now sends cache headers for js/css meaning quicker load times
• Fix for feeders never starting
• Startup sanity check now done in python
• ADSB_URL is checked early in startup and if malformed will cause container startup failure

v2.0 Build 1024

17 June 21

• Container will not fail on docker restart acarshub
• Browser back button now works
• Live Map message box will only show not show non-decoded text if decoded text is present
• Live Map message box will close if open and you click another page link

v2.0 Build 1025

18 June 21

• Reverted docker image to debian:stable-slim to fix Raspbian 32 Bit issues
• Updated SQLAlchemy
• Updated JS/CSS libraries to most recent minor versions

v2.0 Build 1026

18 June 21

• Fixed searches being run if all search boxes are blank
• TS code cleanup regarding some // @ts-expect-error
• Cleaned up commented out old code

Will it be possible to to expose acarsdec options to send acars messages to addr:port via UDP?

This would allow docker-acarshub to send acars messages to programs like FlightAirMap

• Github
• Docker Hub

This is for #33

New features

• Project Eagle
  • Add to system status
  • Health check
  • Websocket feature
• Single socket
• Single page
• Mobile
  • Footer
  • Footer in main body
  • Footer main body text expanded
  • Message tabs scale to display
• jBox
  • Modal for all pages
  • Tooltips
  • Tooltip CSS
• Search page finish modal integration
• Stat page move old sidebar to graphs
• Redirect 404s

Nerd stuff

• CSS cleanup
• HTML cleanup
• CSS lint/standards
• HTML lint/standards
• Python 3.9
• Debian sid-slim image
• Minify JS
• Minify CSS
• Turn on cache
• MD5 file names
• Move all static/template files out of web app and move them in during build
• nginx proxy/gzip
• single JS file in prod mode

Bug fixes

• Highlight alert terms on load and even if dups detected
• Cookie live message page showing correct in modal box
• Alert page modal show correct status
• Fix GitHub docker build
• Fix local docker build
• Fit GitHub ES lint (do we still need this?)

We are currently using bootstrap version 3.4.1 but the current public version is 5.1.3. Currently if we update to v5 it breaks all of the CSS in considerable ways and will require a complete re-think of all of the CSS used in the site.css file.

It would be nice to be on current versions of all dependencies so that dependabot:

a) doesn't bug us about outdated packages
b) incorporate feature enhancements and security issues (not that CSS has security issues, but, as a general rule) are included

Current ACARS/VDLM stats are displayed in the web_improvement branch. I find the generated graphs to be good, but the resolution on it is fairly blocky. Before officially merging this looking at the RRD database setup and how we feed it data should probably be improved:

• I think we need more steps per time interval
• I think we need to feed it data more frequently. Perhaps once a minute?

Right now I am grabbing the data from the acars/vdlm_stats scripts, which run at five minute intervals, and updating the appropriate rrd database. There is a timing issue where (at least on my pis) the script that runs to generate the image tends to run just before the most recent stats are fed in to the database, which causes the generated images to lag by five minutes.

I am considering using python and application.py to update the data. The python rrdtool looks like the ticket, but I had trouble getting it installed in the container. I didn't look in to it hard but it was causing me grief which is why I went down the road of using shell scripts. We could count messages/minute, update the rrd databases, and then generate new images.

Before bothering though, the real issue I haven't fully wrapped my mind around is the resolution in the database needed, and if more frequent updates really solve anything. Before merging to main the database resolution/setup should be finalized so we don't run in to needed to deal with an upgraded db at some point.

12 and 24 hour graphs are identical, I think it is in /docker-acarshub-main/rootfs/etc/services.d/rrd-images/

Under regular graphs should the line be:

rrdtool graph "${OUTPUT_DIR}/12hour.png" --title "12 Hours" --start "-12h" "${ARGS_ALL[@]}" >/dev/null 2>&1

Not a dev so I have no idea how to change it in my local container.

Fritz

Decide what to change the coloring to.

FF and Chrome work fine....

I'm running on ACARS Hub Version: v2.0 Build 1027 and it seems, that searching for tail numbers does not work as expected. I wanted to look for all messages originating from Tail A7-BFS. So I went to search-database -> search for mesages and c&p the tail number at the corresponding field.
That yielded no results - but there must be something, as I got an alert from this tail. Entering the tail manually into the field indicated, that the selector works until I enter the dash. That means, I get everything on A or A7 but nothing for A7-
There are no results for A7BFS as well but: A7 BFS does work (space instead of -).

Update / rewrite helppage.md with a focus on the ADSB stuff

Introduction

This issue will be the roadmap and discussion area for ACARS Hub NextGen and it is a living document as the idea fully materializes. The current state of ACARS Hub, version v2, is a bit of a mixed bag. Functionally, it works well, but the code-base is very messy and difficult to continue improving the code and adding new features. Additionally, container setup is exceptionally difficult for new users and even experienced users re-setting up their system. Finally, development for anyone playing in the code is very slow because (re)building the container takes a very long time because of the non-layered approach under-taken so far.

At the very beginning of this project none of what it has morphed in to was envisioned. I had no idea so many people would download and enjoy ACARS Hub. I was a novice programmer (well, still am mostly, but I'm less green I suppose), and combined with the massive feature increase, design decisions were not thought out in a way that ensured it was "future proof". By having a "clean-slate" approach all of the above issues can be addressed.

At the conclusion of this project ACARS Hub v2 and prior will be End of Life with no further support or bug fixes.

The plan, in high level terms

• Have containers for each individual aspect of ACARS Hub NG (VDLM decoding, ACARS decoding, and the server)
• Rewrite the front end to fully encapsulate a more modular approach
• Create Electron or similar desktop app for viewing of ACARS Hub data

The Plan, in detail

Create docker images of common components

• Create the "SDR Common" base image
• Create a base image with common components for each SDR

Separate Containers

• acarsdec container
• vdlm2dec container
• dumpvdl2 container
• vdlm2dec container
• Remove decoders from main ACARS Hub container
  • Remove dumpvdl2
  • Remove acarsdec
  • Remove build deps of those containers
  • Remove startup scripts
  • Update Dockerfile
  • Update documentation

Back end updates

• Move database code to using "finally" and ensuring all sessions are closed
• Move DB pruning to in-code and not separate process (note: presently pruning in sep process may cause the DB to fail to write incoming messages, or fail to run a query, because the DB is locked outside of the main app. SQLite is not meant to deal with multiple processes accessing/writing data so we have to move this back under the watch of SQLAlchemy to avoid issues)
• Move FTS creation to in code and not a separate startup process
• Remove cont-init DB processing

Front end rewrite

Application

Misc

• Update GitHub Actions to use new acarshub-ng tag
• Set up linters for both pre-commit hooks and the CI on GitHub to be in sync
• Contributors/dev document

Version 2.2.0

Version 2.2.0 changes the database structure slightly in two ways:

1. The time stamp of ACARS messages are no longer stored as strings and instead are stored as numbers
2. The first startup, as it completes the migration to the new database structure, will take a while. Log messages may be slow trickling out as it completes the tasks. THIS IS OKAY! Please let it do it's thing.

These changes are part of a process necessary to improve database and startup performance.

PLEASE READ THE FOLLOWING CLOSELY!!!

Please read the intro :)

By upgrading to 2.2.0 or later you will NOT be able to downgrade to any ACARS Hub build lower than 1045 or version 2.1.4. If the upgrade fails and you back up the database (method below) we can recover the data and give it another try. Otherwise, nuking the database and starting fresh will be your only option.

I've tested this out to ensure it works but there is a possibility I've messed up somewhere and your database will be corrupted. If you want to ensure continuity of data in case it doesn't work, like your saved messages, alert counts, frequency counts, etc, please do one or both of the following:

Add the following to your ACARS Hub docker-compose.yaml configuration

    environment:
      - BACKUP_THE_DB=True

This will cause the container to create a backup of your current database file before modifying it. Additionally, or if you prefer a manual backup instead of letting the container do it:

sudo su -
cd /var/lib/docker/volumes/ubuntu_acars_data/_data
cp messages.db messages-backup.db

You might need to change the path to the docker volume containing your ACARS Hub data, but it will be in /var/lib/docker/volumes/.

Upgrading the database

docker pull fredclausen/acarshub:test-pr
docker-compose up -d
docker logs -f acarshub

Please watch the logs. This process may take more than 10 minutes depending on different factors. You are looking for something like the following:

[cont-init.d] 02-database: executing... 
[02-database] 2021/07/10 20:48:01 Checking to see if database needs upgrades
[02-database] 2021/07/10 20:48:02 Converting time from string to number
[02-database] 2021/07/10 20:48:02 Renaming current tables
[02-database] 2021/07/10 20:48:02 Creating new tables
[02-database] 2021/07/10 20:48:02 Copying old data
[02-database] 2021/07/10 20:48:04 Dropping old tables
[02-database] 2021/07/10 20:54:38 Updating FTS cache. May take a while
[02-database] 2021/07/10 20:58:31 Ensuring no columns contain NULL values
[02-database] 2021/07/10 20:58:33 done with de-nulling
[02-database] 2021/07/10 20:58:33 Adding text index
[02-database] 2021/07/10 20:58:46 Adding icao index
[02-database] 2021/07/10 20:58:47 Adding flight index
[02-database] 2021/07/10 20:58:49 Adding tail index
[02-database] 2021/07/10 20:58:50 Adding depa index
[02-database] 2021/07/10 20:58:55 Adding dsta index
[02-database] 2021/07/10 20:58:56 Adding msgno index
[02-database] 2021/07/10 20:58:58 Adding freq index
[02-database] 2021/07/10 20:59:00 Adding label index
[02-database] 2021/07/10 21:00:52 Completed upgrading database structure
[cont-init.d] 02-database: exited 0.

Once you see the completed message that means the upgrade was successful. Please wait until you see

[webapp] 2021/07/10 21:03:23 [database]: Pruned main database of x records
[webapp] 2021/07/10 21:03:23 [database]: Pruned alerts database of x records

You might see a couple of CRITICAL WORKER TIMEOUTS. I think I've figured out the cause, but in case you do see them PLEASE BE PATIENT; it will start up. Also, please copy the startup log with the critical worker timeouts and open an issue here on GitHub or let me know about it on Discord.

Once you see the Pruned messages that means everything started up properly. If you added BACKUP_THE_DB to your yaml file, please remove it at this time.

Subsequent restarts of the container should not do any upgrading of the database and should be very quick. My test container went from more than a minute from the time the container started up until the startup process was done to doing it in less than 15 seconds with these (and other) improvements made to the startup sequence.

IF YOU USE DB_BACKUP

Most users this will not apply to. If you don't know what DB_BACKUP is you can skip this.

You will need to manually update your backup database BEFORE running the update in your container. Update the msg_time column type to INTEGER or whatever is appropriate for your type of database. SQLite, used in the container, is using INTEGER.

Remove the "strobe" affect

• Git fetch updates on container start to catch updates
• Load those files from disk

Watching szpajder/dumphfdl#21 and will implement when feature is complete

Could possibly tweak the ADSB part of readme??

from:

"If you desire enhanced ADSB and ACARS message matching, and are running Mike's tar1090 container, you can enable the following option:"

to

"If you desire enhanced ADSB and ACARS message matching and thus show coloured aircraft icons on Live Map, and are running Mike's tar1090 container, you can enable the following option:"

Spent a while wondering why I had a lot of only Blue a/c and messages coming in. Queried and followed wiedehopf's polite R.T.F.M. suggestion. Having originally read over that I section I figured that it would only enhance the info displayed when hovering over the a/c and did not realise it would also link in the ACARS messages.

Just a suggestion.