If I understand haulage correctly, it logs all the bytes sent/received by a user and updates the customer table every (interval) seconds. I think it'd be super useful to enable some way for us to tease out aggregate information (e.g. bytes consumed in last 5mins/10mins/30mins/hour); both for individual users as well as the whole system. Not the most urgent thing on earth but definitely a "nice to have" item.

Due to the location of the enforcement rateliming queues and the current simple matching approach, local download traffic receives the same ratelimiting as remote downloading traffic when ratelimiting is enabled, and local traffic which does not leave the routing host is not ratelimited at all if a ratelimit is configured.

Addressing this will require a pretty large change to the ratelimiting architecture to correctly classify and police/limit all 4 types of traffic UL/DL X Backhaul/Local.

Currently the packet processing subsystem is shutdown violently, leading to data loss during the last interval. A clean shutdown could ensure all packets are appropriately logged to disk before terminating.

Right now haulage assumes each user is represented by only a single IP. This could allow "double spending" across the IPs before the database synchronization period. Multiple IP addresses per-user is important for supporting dual IPv4/IPv6 connectivity.

The current code assumes that all users will be in the database on startup, and fails to create new qdiscs for users that are later added to the system while operating. As a temporary workaround you can add new users and then restarting haulage to pick up the changes. The system does respond correctly to changes in the user configuration, and will successfully pick up changes to existing users without a restart.

With uw-ictd/colte#106, the colte database model is responsible for the currencies table in the shared database. Unfortunately, I made a rookie mistake and re-used the same table name in the colte project, which is expected in many legacy deployments to connect to the same database as haulage. This creates an issue where the currencies table may already be present in the database if any of the colte services have run, causing the early haulage migrations to fail. Since sqlx applies all migrations iteratively, the early migrations creating the currencies table, which is later dropped by haulage, will conflict and cause the migration chain to fail. If all haulage migrations are run first there is no issue, but this cannot be guaranteed in real world deployment.

Haulage requires libpcap-dev to compile; for some reason this package is not included on the standard Debian and Ubuntu packages. We should add it to a build dependencies target (preferable) or note it in README (less preferable).

Node 10 is the default system node version in the recommended debian buster platform, but the webgui fails to run with it. We should either suggest an alternative way to install an up-to-date version of node, or fix the backwards compatibility issue.

Error log:

Jun 11 13:58:11 cobble haulage[29636]: thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: UserLookupError', haulage/src/accounter.rs:100:65
Jun 11 13:58:11 cobble haulage[29636]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Jun 11 13:58:11 cobble haulage[29636]:   aggregator: user
Jun 11 13:58:11 cobble haulage[29636]:    aggregation: 10.45.1.8
Jun 11 13:58:11 cobble haulage[29636]:     Jun 11 20:58:11.998 ERRO Failed to initialize reporter, error: Failed to lookup user, id: 10.45.1.8
Jun 11 13:58:12 cobble haulage[29636]:   accounter: user
Jun 11 13:58:12 cobble haulage[29636]:    Jun 11 20:58:12.004 ERRO Failed to dispatch, error: channel closed

One of the communities outside Oaxaca apparently uses a monthly membership model, and I think it'd be a good feature to add to haulage. Generally speaking I don't want to make haulage too feature-heavy, would prefer integrate with existing WISP management tools than race them. Compared to these tools, the biggest value-adds haulage provides are (1) it supports prepaid customers (2) its simple and (3) its free. I think/hope we could implement this with minimal logic - a database field to mark whether the user is on a monthly or data plan, and then some logic to automatically draw-down from their $ balance every month, mark how long they're paid through, and cut them off if they ever go below 0.

There are several aministrative tasks like updating the database schema or adding new users that could benefit from a standalone scriptable admin tool. Right now some of this functionality is implemented with python scripts, but it would be good for all functionality that touches the database to use the same underlying SQLX definitions.

Comments in the config file should explain how the file is structured and what typical values are of the configuration options themselves. This will make it easier to understand what's going on in the file and for users to make the changes they need to make!

The haulagedb tool does not correctly handle the foreign key constraint to allow dropping subscribers with existing usage history.

There is a hack in the code right now that abuses the logging api.

Jun 11 21:58:44.434 ERRO Unable to update policy for zero balance sub, error: Database operation failed: error returned from database: column "subscriber_id" does not exist

Right now haulage uses a naive userspace packet monitoring approach that involves additional packet copying. This has the benefit of placing few dependencies on the underlying system, but comes at the cost of performance. With LEO satellite use-cases looming for network-in-a-box community networks, it would be good to support an alternative traffic monitor more tightly integrated with something like dpdk/vpp, or the kernel conntrack system for performance up to the ~1gpbs range.

haulage

INFO[0000] Starting haulage
INFO[0000] Parsed parameters Parameters="{20m0s 1m0s enp1s0 {5s colte_db colte_db colte_db}}"
INFO[0000] Initializing context
ERRO[0000] Unable to query initial bridged state error="Error 1146: Table 'colte_db.customers' doesn't exist"

The current architecture has no unit tests. This is awful and will not allow the project to grow.

Refactor all configuration out of the source code and into a configuration file or command line flags.

In integrating with the CoLTE database, the raw_up and raw_down DB columns by design should be cumulative sums, not just over the interval.

When nat is enabled, the source ip address seen by the tc egress qdisc is already updated to be the local address, meaning simple matching rules based on the source IP are not sufficient to map egress flows to queues.

Should follow best practices for open source projects.