jow- / nlbwmon Goto Github PK

Let's look solely on the wlan0 (2.4 gHz) bandwidth, which is used by only one device, a Satellite laptop.

In nlbwmon's info, the Satellite's total bandwidth is less than 300 MB.
But in vnstat, wlan0's total bandwidth is 1.42 GB.

Can someone kindly help me understand the discrepancy?
Please know that both have the same billing cycle (same cutoff date).

Dear @jow-

A few people who have installed nlbwmon have reported issues with the tracking of ipv6 on a thread in the Lede forums.

A user named hnyman

And another user named philjohn

And a third user named moeller0

I'm trying to run this program on Ubuntu 20.04 Beta but got empty result from sudo nlbw -c show.
I installed conntrack (sudo apt-get install conntrack) and confirm nf_conntrack is loaded (lsmod | grep nf_conntrack).
I set net.netfilter.nf_conntrack_acct to 1 according to #35, but still got empty result.
My IP address is 192.168.213.128/24 and I run nlbwmon with sudo nlbwmon -s '192.168.0.0/16'.
Did I miss something?

As per the title. If an application (say DNS or in this case Mumble) uses multiple protocols (both TCP and UDP), then this will result in multiple records in the output.

Intuitively - and based on the contents of protocols.txt - I would have expected traffic for port/protocol pairs that had the same application name to be aggregated together.

Dear folks,

About a week ago, I installed nlbwmon on my router (Lede 17.01.2 is firmware) with the hopes to have a log of my bandwidth use. (I simply need to see how much bandwidth I've used for each billing month).

Is there any reason why I should still use vnstat? In other words, is there something that vnstat does and which I need that nlbwmon doesn't do?

Thank you.

luci-app-nlbwmon version: git-17.217.44847-21ae7f2-1
nlbwmon version | 2017-08-02-32fc0925-1

Hi guys,

I'm a newbie with a WD MyNet 750 router running LEDE Reboot 17.01.2 r3435-65eec8bd5f. I have a 256MB USB thumbdrive plugged into the back of the router. It's formatted as fat32.

I need to see, in every billing cycle, how much bandwidth I'm using. The billing cycle begins on the 10th of every month. I want to keep a record of all my months forever (In other words, I don't want to lose my stats for the July billing cycle when the August billing cycle is complete).

@hnyman suggested that I use nlbwmon (and luci-app-nlbwmon) to accomplish this goal.

My Netlink Bandwidth Monitor - Configuration > Advanced Settings looks like this:
http://i.imgur.com/vQtku88.png

Maximum entries: 0 (I changed it from 10000 for now. With a 256MB USB thumbdrive, I'm not worried about storage space)
Stored periods: 0 (coz I wanna keep databases forever. Maybe later, I'll change this. But since I have a huge storage space -- a 256MB thumbdrive -- I can afford "forever")
Commit Interval: 60s (at least for now, as I make sure everything is working. Later on, after I verify smooth operations, I'll dial it back to maybe 10 minutes)
Refresh Interval: 30s (the default setting)
Database directory: "/mnt/sda1/" (this is my 256MB USB thumbdrive)

Some questions:

1. nlbwmon can accomplish my goal(s), right?
2. How can I view the logs?

https://github.com/kiougar/luci-wrtbwmon seems to support it but doesn't seem to be maintained anymore. 🙂

hi jow, a quick question.

i've had a little look but i'm not sure what the best way would be of accessing data from nlbwmon to process in code and send to a database on a different host on the internet. if possible i'd like to store nothing on the persistent storage on my router.

is there a good hook in nlbwmon that i can use?

Probably a bit of an edge case, but due to my ISP's flakiness lately, my machines have had their IPv6 addresses changed frequently. Because of this, the popup on the Traffic Distribution tab of the luci app is impossible to read. Is there any way to pop it out into its own window, or make it scrollable otherwise?

When you use a bridge, the traffic is correctly add to the database but with Mac Adress = unknown...
So all traffic is in one item, and can only be shown in a popup...
Inside the popup the informations looks like correct...

Can we have a parameter, options or a modification of the main display page to have all the "other" MAC shown directly ?

Thanks...

When grouped the data by protocols, I found the following traffic ranged at the top

According to /etc/protocols the No.1 traffic is protocol IP or HOPOPT.
I wonder what kind of traffic it exactly is and what it should be classified in layer7 column?

In my case when I used OpenVPN v2ray on the router the accounting also count the router. So the total would be doubled, and half of it was the VPN connection I guess.
So I want to exclude this VPN connection or maybe the router itself from accounting. I tried playing with the subnet since the router IP that showed on nlbwmon wasn't from the same subnet but from isp. But no luck so far.

picrel:

Error information:
make[3]: Entering directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e' make[4]: Entering directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e'
make[5]: Entering directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e' Scanning dependencies of target nlbwmon make[5]: Leaving directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e'
make[5]: Entering directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e' [ 9%] Building C object CMakeFiles/nlbwmon.dir/client.c.o [ 18%] Building C object CMakeFiles/nlbwmon.dir/database.c.o [ 27%] Building C object CMakeFiles/nlbwmon.dir/neigh.c.o In file included from /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/netlink.h:30:0, from /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/msg.h:15, from /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e/neigh.c:29: /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/handlers.h:134:19: error: 'struct ucred' declared inside parameter list [-Werror] struct ucred **); ^ /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/handlers.h:134:19: error: its scope is only this definition or declaration, which is probably not what you want [-Werror] /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/handlers.h:208:36: error: 'struct ucred' declared inside parameter list [-Werror] unsigned char **, struct ucred **)) ^ /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/handlers.h: In function 'nl_cb_overwrite_recv': /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/handlers.h:210:17: error: assignment from incompatible pointer type [-Werror=incompatible-pointer-types] cb->cb_recv_ow = func; ^ In file included from /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/msg.h:15:0, from /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e/neigh.c:29: /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/netlink.h: At top level: /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/netlink.h:57:13: error: 'struct ucred' declared inside parameter list [-Werror] struct ucred **); ^ In file included from /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e/neigh.c:29:0: /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/msg.h:52:16: error: field 'nm_creds' has incomplete type struct ucred nm_creds; ^ /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/msg.h: In function 'nlmsg_set_creds': /home/everun/build/lede/trunk/source/staging_dir/target-arm_cortex-a9_uClibc_eabi/usr/include/libnl-tiny/netlink/msg.h:207:39: error: dereferencing pointer to incomplete type 'struct ucred' memcpy(&msg->nm_creds, creds, sizeof(*creds)); ^ cc1: all warnings being treated as errors make[5]: *** [CMakeFiles/nlbwmon.dir/neigh.c.o] Error 1 make[5]: Leaving directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e'
make[4]: *** [CMakeFiles/nlbwmon.dir/all] Error 2
make[4]: Leaving directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e' make[3]: *** [all] Error 2 make[3]: Leaving directory /home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e'
make[2]: *** [/home/everun/build/lede/trunk/source/build_dir/target-arm_cortex-a9_uClibc_eabi/nlbwmon-2017-07-28-76487b5e/.built] Error 2
make[2]: Leaving directory /home/everun/build/lede/trunk/source/feeds/packages/net/nlbwmon' make[1]: *** [package/feeds/packages/nlbwmon/compile] Error 2 make[1]: Leaving directory /home/everun/build/lede/trunk/source'
make: *** [package/feeds/packages/nlbwmon/compile] Error 2

Running for a few days and I get this in syslog when the GUI shows 'No data recorded yet.' and 'Force reload' for MAC data:

Wed Jan 17 15:17:55 2018 daemon.err nlbwmon[9277]: Unable to handle 'dump' command: Resource temporarily unavailable
Wed Jan 17 15:17:55 2018 daemon.err uhttpd[1293]: Error while processing command: No data available
Wed Jan 17 15:17:59 2018 daemon.err nlbwmon[9277]: Unable to handle 'dump' command: Resource temporarily unavailable
Wed Jan 17 15:17:59 2018 daemon.err uhttpd[1293]: Error while processing command: No data available

running under console completes fine:

./usr/sbin/nlbwmon dump
(lots of data)

Have relocated storage to /opt onto removable USB which is accessible and fine.

Was using nlbwmon on openwrt for years but missing it
on plain linux router/firewall installs ( when openwrt does not cut it )

Ntopng seems like major overkill for simple bandwith/host tracking.
wrtbwmon relies on extra iptables rulles (which i like nice and tidy, and manageable)
(just want to track all hosts being routed trough the firewall)

Was trying to compile nlbwmon for devuan (debian without systemd) .
But got stuck on avl.h and usock.h
Which seem to be parts of libubox (openwrt pkg only).

avl.h can be found with libavl-dev package but not usock.h

So was wondering if you could just use avl.h and usock.h from x86openwrt on
regular linux or any other solution that would make nlwbmon work on non openwrt linux.

Thanx

Hello, I notice that SSDP traffic, coming from UPnP and NAT-PMP, on UDP port 1900, is counted.
It should not, since it is local traffic sent in broadcast (239.255.255.250 and ff02::c).

I'm getting the following errors every few hours:
daemon.err nlbwmon[21812]: Netlink receive failure: Out of memory
daemon.err nlbwmon[21812]: Unable to dump conntrack: No buffer space available

My stats:
# cat /proc/sys/net/netfilter/nf_conntrack_count 632
# cat /proc/sys/net/netfilter/nf_conntrack_max 16384

I've already increased buffer size to 100M and restarted nlbwmon but still getting these errors. Also changed the Commit interval to 1hr, hoping that this commit will flush the buffer after writing to the database but not helping. Strange this is, even after increasing the buffer to 100M and running for a few hrs and getting this error, my total memory usage still shows the same. I'd think it will increase by 100M?? How to fix this? Thanks.

1.Have you considered supporting Prometheus
2.What is the lowest kernel version

I'm using this software on a router with OpenWRT but I can't log the traffice to/from the internet. This is my configuration:

config nlbwmon
option commit_interval '24h'
option refresh_interval '30s'
option database_directory '/var/lib/nlbwmon'
option database_generations '10'
option database_interval '1'
option database_limit '10000'
option protocol_database '/usr/share/nlbwmon/protocols'
list local_network '0.0.0.0/0'
list local_network 'bridge'

eth0 and eth1 are configured with a DHCP bridge.
Do I have to specify individual interfaces or do I need to specify the bridge?

Hi,

I've been running nlbwmon for a few hours but I'm missing many hosts that do not show up.
For example I have video cams that constantly transmits data to a server on the Internet but they do not show on nlbwmon. I can see the hosts in conntrack and I can also see that the counters are constantly being reset while the same connection is being kept so it looks like nlbwmon does read the data. I'm not sure how nlbwmon works, but maybe you only add the stats once the connection is ended? In my case the cam uses the same connection. Any tips on how to debug this?

Thanks.

I want to categorize the traffic in others but I don't have any info of it.
can you add a feature that shows the most used protocol (at least port and maybe ip too) in other category?
or is that impossible to track?

this way I can label correctly the traffic in other into correct protocol and know what is using how much.

Downloading https://openwrt.proxy.ustclug.org/releases/18.06.8/packages/mipsel_24kc/luci/luci-app-nlbwmon_git-20.213.35591-16dfb55-1_all.ipk
Collected errors:

• satisfy_dependencies_for: Cannot satisfy the following dependencies for luci-app-nlbwmon:

•  kernel (= 5.4.60-1-d38eb031bb825d2536631cceb1dff1fa)
  

• opkg_install_cmd: Cannot install package luci-app-nlbwmon.

The readme.md on https://github.com/jow-/nlbwmon says

If the user is not interested in layer7 protocol information, removing all protcools from the protocol file will classify all traffic into 'other', vastly reducing the database size.

I'd love to save on database size. So what exactly is Layer7 protocol? And more importantly, how do I know if I'm using that protocol?
I surf the web. If it helps, I don't torrent or Netflix or play online games or whatever.

is it possible to disable ipv6 in statistics ?
I don't have any ipv6 connectivity on WAN side.
and I want to log the WAN side only.

also I have lede installed on a router and the router is connected to internet via bridge to a modem.
how do I log the traffic consumption for internet?
the router has the physial interfaces as eth0 and eth1.
eth1 is lan and eth0 in the Ethernet that I have connected as bridge to modem(so that I wouldn't have to port forward on the modem too and all that.)
should I just disable bridge and connect via lan to modem? that way the internet traffic accounting would be easier?

Hi!

Right now Openwrt is switching from iptables to nftables and netlink is supposed to be universal, not depending on iptables or nftables since it is an api for netfilter (even though it's dependencies are incorrect on openwrt).
I have tried bwmon on an nftables router and even though connections are correctly shown in Status > Realtime Graphs > Connections, in the bwmon luci page only connections from the router itself are shown and updated.

Support needs to be added with a change to this repo?

Thanks!

I've been using this code base to create something more comprehensive (and currently annoying) as a network monitor for my personal use. Basically sending push notifications when some IOT device sends data to an unexpected country or just sends a lot of data which I don't expect it to.

In the process of hacking this code, I found two major bugs which I think are worth fixing. Unfortunately I don't have a pull request but these are easy to fix:

1. I think nfnetlink_dump() does not work as intended.

for (err = 1; err > 0; ) {
	ret = nl_recvmsgs(nl, cb);
	if (ret == 0) {
		err = 0;
		break;
	}
	else if (ret < 0) {
		fprintf(stderr, "Netlink receive failure: %s\n", nl_geterror(ret));
		err = (-ret == NLE_NOMEM) ? -ENOBUFS : -EIO;
		break;
	}
}

nl_recvmsgs() returns 0 when successful so this loop only runs once. Thus this dump only processes the first entry. It is necessary to keep calling nl_recvmsgs() until the done / error callback are called which will set the err value and exit the loop (or if ret < 0).

I have also changed NL_SKIP to NL_OK in handle_dump().

I think it's also worth running nfnetlink_dump(allow_insert=true) when the daemon starts, otherwise existing connections started before the daemon will not be added to the DB (which will only try to update the DB and fail).

2. There are some avl_tree cmp functions returning a direct subtraction of uint32_t which are used for the result of the signed int cmp function. So (uint32_t)3000000000 - (uint32_t)1 will return a negative value when the function returns.

Also note that (uint32_t)1 - (uint32_t)2 does not return a negative value although by casting it back to (signed int) the result will be negative.
memcmp(a->u32, b->u32, sizeof(a->u32)) works fine for all cases (at least for host order values which most of the values are).

how can I configure the setting to get correct torrent traffic?
I have added the port set in deluge torrent client in nlbwmon but an small traffic is logged as deluge-tcp and most is counted as others.
the port mapping I've put is
"6 1803 deluge-tcp"
"17 1803 deluge-udp"
should I have put something else?

is there anyway to see the traffic that is counted as other in port ip protocol format? so that I can classify them easily?

I tried to use nlbwmon on a "fully-fledged" Linux system, with glibc, libnl, etc (Arch Linux x86_64).

However, it either does nothing at all or crashes almost immediately, for instance:

# nlbwmon -i 10s -r 1s -s 172.23.184.64/26
Interval: 20170901 - Usage: 40/12840 Bytes                                          

*** stack smashing detected ***: <unknown> terminated                               
Aborted (core dumped)

This is on a laptop: 172.23.184.64/26 is my local network, connected on wlan0. When I pass -s 0.0.0.0/0 nlbwmon does not crash, but it seems to do nothing.

Here is a backtrace of the crash:

#0  0x00007f1a6908d8a0 in raise () from /usr/lib/libc.so.6                          
#1  0x00007f1a6908ef09 in abort () from /usr/lib/libc.so.6                          
#2  0x00007f1a690d0517 in __libc_message () from /usr/lib/libc.so.6                 
#3  0x00007f1a691606bf in __fortify_fail_abort () from /usr/lib/libc.so.6           
#4  0x00007f1a69160682 in __stack_chk_fail_local () from /usr/lib/libc.so.6         
#5  0x000055f7ab783ab5 in ipaddr_parse (msg=0x55f7ad886770, arg=0x7ffe41144320) at /home/zorun/tmp/nlbwmon/neigh.c:190                                                  
#6  0x00007f1a6983f5f9 in nl_recvmsgs_report () from /usr/lib/libnl-3.so.200        
#7  0x00007f1a6983fa29 in nl_recvmsgs () from /usr/lib/libnl-3.so.200               
#8  0x000055f7ab783bc1 in ipaddr_to_ifindex (family=2, addr=0x7ffe41144384) at /home/zorun/tmp/nlbwmon/neigh.c:213                                                      
#9  0x000055f7ab783ee5 in update_macaddr (family=2, addr=0x7ffe41144450) at /home/zorun/tmp/nlbwmon/neigh.c:302                                                         
#10 0x000055f7ab7847d2 in parse_event (reply=0x55f7ad886670, len=164, allow_insert=false, update_mac=true) at /home/zorun/tmp/nlbwmon/nfnetlink.c:204                   
#11 0x000055f7ab7849b1 in handle_dump (msg=0x55f7ad8808d0, arg=0x7ffe411445ec) at /home/zorun/tmp/nlbwmon/nfnetlink.c:243                                               
#12 0x00007f1a6983f5f9 in nl_recvmsgs_report () from /usr/lib/libnl-3.so.200        
#13 0x00007f1a6983fa29 in nl_recvmsgs () from /usr/lib/libnl-3.so.200               
#14 0x000055f7ab784ca5 in nfnetlink_dump (allow_insert=false) at /home/zorun/tmp/nlbwmon/nfnetlink.c:381                                                                
#15 0x000055f7ab7850fe in handle_refresh (tm=0x55f7ab98a960 <refresh_tm>) at /home/zorun/tmp/nlbwmon/nlbwmon.c:154                                                      
#16 0x00007f1a69a53c30 in uloop_run_timeout () from /usr/lib/libubox.so             
#17 0x000055f7ab784d4a in uloop_run () at /usr/include/libubox/uloop.h:111          
#18 0x000055f7ab785953 in server_main (argc=7, argv=0x7ffe41144908) at /home/zorun/tmp/nlbwmon/nlbwmon.c:364                                                            
#19 0x000055f7ab7859cb in main (argc=7, argv=0x7ffe41144908) at /home/zorun/tmp/nlbwmon/nlbwmon.c:377  

Full backtrace:

#5  0x000055f7ab783ab5 in ipaddr_parse (msg=0x55f7ad886770, arg=0x7ffe41144320) at /home/zorun/tmp/nlbwmon/neigh.c:190
        hdr = 0x55f7ad8867c0
        ifa = 0x55f7ad8867d0
        addr = 0x55f7ad8867e0
        tb = {0x0, 0x55f7ad8867d8, 0x55f7ad8867e0, 0x55f7ad8867e8, 0x0, 0x0, 0x55f7ad8867f8, 0x0, 0x55f7ad8867f0}
        query = 0x7ffe41144320
#6  0x00007f1a6983f5f9 in nl_recvmsgs_report () from /usr/lib/libnl-3.so.200                                                                                           
No symbol table info available.                                                                                                                                        
#7  0x00007f1a6983fa29 in nl_recvmsgs () from /usr/lib/libnl-3.so.200                                                                                                  
No symbol table info available.                                                                                                                                        
#8  0x000055f7ab783bc1 in ipaddr_to_ifindex (family=2, addr=0x7ffe41144384) at /home/zorun/tmp/nlbwmon/neigh.c:213                                                     
        query = {family = 2, addr = 0x7ffe41144384, ifindex = 0}                                                                                                       
        ifa = {ifa_family = 2 '\002', ifa_prefixlen = 0 '\000', ifa_flags = 0 '\000', ifa_scope = 0 '\000', ifa_index = 0}                                             
        msg = 0x55f7ad886720                                                                                                                                           
#9  0x000055f7ab783ee5 in update_macaddr (family=2, addr=0x7ffe41144450) at /home/zorun/tmp/nlbwmon/neigh.c:302                                                        
        ptr = 0x55f7ab98a6f0 <subnets>                                                                                                                                 
        key = {u32 = {2, 1622677420, 0, 0, 0}, data = {family = 2 '\002', addr = {in = {s_addr = 1622677420}, in6 = {__in6_u = {                                       
                  __u6_addr8 = "\254\027\270`", '\000' <repeats 11 times>, __u6_addr16 = {6060, 24760, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {1622677420, 0, 0, 0}}}}}}     
        res = 0x0                                                                                                                                                      
        ifindex = 2   
#10 0x000055f7ab7847d2 in parse_event (reply=0x55f7ad886670, len=164, allow_insert=false, update_mac=true) at /home/zorun/tmp/nlbwmon/nfnetlink.c:204                  
        hdr = 0x55f7ad886670                                                                                                                                           
        gnlh = 0x55f7ad886680                                                                                                                                          
        attr = {0x0, 0x55f7ad886684, 0x55f7ad8866b8, 0x55f7ad8866ec, 0x0, 0x0, 0x0, 0x55f7ad8866f4, 0x55f7ad8866fc, 0x0, 0x0, 0x55f7ad88670c, 0x55f7ad886704,          
          0x0 <repeats 12 times>}                                                                                                                                      
        tuple = {0x0, 0x55f7ad8866bc, 0x55f7ad8866d0, 0x0}                                                                                                             
        counters = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}                                                                                                                      
        r = {family = 2 '\002', proto = 0 '\000', dst_port = 0, src_mac = {ea = {ether_addr_octet = "\000\000\000\000\000"}, u64 = 0}, src_addr = {in6 = {__in6_u = {  
                __u6_addr8 = "`\270\027\254", '\000' <repeats 11 times>, __u6_addr16 = {47200, 44055, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {2887235680, 0, 0, 0}}}, in = { 
              s_addr = 2887235680}}, count = 0, out_pkts = 0, out_bytes = 0, in_pkts = 0, in_bytes = 0, node = {list = {next = 0x0, prev = 0x0}, parent = 0x0,         
            left = 0x0, right = 0x0, key = 0x0, balance = 0 '\000', leader = false}}
        orig_saddr = {__in6_u = {__u6_addr8 = "`\270\027\254", '\000' <repeats 11 times>, __u6_addr16 = {47200, 44055, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {2887235680,   
              0, 0, 0}}}                                                                                                                                               
        orig_daddr = {__in6_u = {__u6_addr8 = "\032\225\340[", '\000' <repeats 11 times>, __u6_addr16 = {38170, 23520, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {1541444890,   
              0, 0, 0}}}                                                                                                                                               
        reply_saddr = {__in6_u = {__u6_addr8 = "\032\225\340[", '\000' <repeats 11 times>, __u6_addr16 = {38170, 23520, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {1541444890,  
              0, 0, 0}}}                                                                                                                                               
        reply_daddr = {__in6_u = {__u6_addr8 = "`\270\027\254", '\000' <repeats 11 times>, __u6_addr16 = {47200, 44055, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {2887235680,  
              0, 0, 0}}}                                                                                                                                               
        orig_pkts = 0                                                                                                                                                  
        orig_bytes = 0                                                                                                                                                 
        reply_pkts = 0                                                                                                                                                 
        reply_bytes = 0                                                                                                                                                
#11 0x000055f7ab7849b1 in handle_dump (msg=0x55f7ad8808d0, arg=0x7ffe411445ec) at /home/zorun/tmp/nlbwmon/nfnetlink.c:243                                              
        hdr = 0x55f7ad886670                                                                                                                                           
        allow_insert = 0x7ffe411445ec                                                                                                                                  
#12 0x00007f1a6983f5f9 in nl_recvmsgs_report () from /usr/lib/libnl-3.so.200                                                                                           
No symbol table info available.
#13 0x00007f1a6983fa29 in nl_recvmsgs () from /usr/lib/libnl-3.so.200
No symbol table info available.
#14 0x000055f7ab784ca5 in nfnetlink_dump (allow_insert=false) at /home/zorun/tmp/nlbwmon/nfnetlink.c:381                                                               
        req = 0x55f7ad880880                                                                                                                                           
        cb = 0x55f7ad882570                                                                                                                                            
        tuple = 0x55f7ad881574                                                                                                                                         
        ip = 0x55f7ad881578                                                                                                                                            
        proto = 0x55f7ad881578                                                                                                                                         
        hdr = {nfgen_family = 0 '\000', version = 0 '\000', res_id = 0}                                                                                                
        err = -11                                                                                                                                                      

on openwrt, 18.06.1, 18.06.2, wrt32x:
no matter what the accounting period is, the diplay of host activity only shows the curretn accounting period.

On startup, nlbwmond spams syslog with 'unable to delete database' type messages. I'm using compressed .gz databases, so it's correct that a plain .db doesn't exist, but the .gb.gz file does.

I have set accounting period to fixed interval with start date 2022-01-01 and interval 1 days. It should reset the accounting period at 12AM every day.
But i have noticed, it counts the accounting period from the last commit to database.
I restart the router every morning at 6AM. And before restart, i run,

/usr/sbin/nlbw -c commit

So the accounting period for that day is counted after the commit at 6AM.

Can i somehow set accounting period from 12AM to 12AM next day?

can I specify a port range instead of one port?

Hello, I have a stable luci/openwrt router (https://openmptcprouter.com), its uptime is rock-solid (>8days, I sometime reboot the instance, or upgrade it).
But as soon as I install nlbwmon (x86_64, via opkg install nlbwmon), the uptime goes crazy : 2 days, then 1 day, then 18 hours, then 4 hours, then 30min....).
I don't have anything in the logs, but I only have "system" logs persistent, I can't keep kernel logs.... but as the router reboots itself, there must me something wrong.

maybe the kernel (conntrack) settings, due to the regular polling that the nlbwmon daemon does to it ?
Could anyone give me some "safe" settings to put via sysctl ?
for example, here are my current settings :

net.netfilter.nf_conntrack_acct = 1
net.netfilter.nf_conntrack_buckets = 4096
net.netfilter.nf_conntrack_checksum = 0
net.netfilter.nf_conntrack_count = 3125
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 64
net.netfilter.nf_conntrack_frag6_high_thresh = 4194304
net.netfilter.nf_conntrack_frag6_low_thresh = 3145728
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_no_window_check = 1
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 7440
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.nf_conntrack_max = 131072

With OpenWrt 22.03.0-rc0, nwlbmon does not show any useful information. It mainly tracks upload connections from the OpenWRT router itself, but only a few download connections from the network devices. For example, after a Fire TV session, it shows 45.13 MiB upload for the stick, but only 29.85 KiB download - should be more than a few gigabytes.

Config file:

config nlbwmon
        option netlink_buffer_size '524288'
        option database_interval '1'
        option protocol_database '/usr/share/nlbwmon/protocols'
        option refresh_interval '60s'
        option database_directory '/mnt/sda1/data/nlbwmon'
        option commit_interval '10m'
        option database_generations '60'
        list local_network '192.168.0.0/16'
        list local_network '172.16.0.0/12'
        list local_network '10.0.0.0/8'
        list local_network 'guest'
        list local_network 'lan'
        option database_limit '10000'
        option database_compress '1'

In "Netlink Bandwidth Monitor - Configuration"/"Local interfaces", which should be selected?

Lan [the only one selected by default]
Wan
Wan6

If there's any other info that would help you help me, please let me know.

When trying to view data for prior accounting periods, the graphs in LuCI, the graphs just re-animate but the data on the page does not change. In the system log, this message appears:

Sat Jan 12 16:53:07 2019 daemon.err nlbwmon[10274]: Unable to handle 'dump' command: No error information Sat Jan 12 16:53:07 2019 daemon.err uhttpd[1592]: Error while processing command: No data available

The databases are still there and can be viewed with the 'nlbw' command. This happens on both mvebu and x86 hardware, in both 17.01 and 18.06.

When running any of the nlbw commands to see the connections, the mac addresses of my router's connected devices all show up as zeroes. Is there a fix to that?

Let's say my nlbwmon program is happily doing daily backups of the bandwidth for August 1 through 15 to the USB stick inserted on the back of the router.

On August 20, there is a power outage to the city and the router abruptly loses power. There is no power for a few days (Aug 20 through 24). On Aug. 25, power is restored and the router is powered on. What happens from Aug. 25 onwards? When nlbwmon tries to backup to the USB stick, will it overwrite the data from Aug. 1 through 15? In other words, will the USB stick now contain just August 25 onwards?

Or will the USB stick -- this is my hope -- contain Aug 1-15 and Aug. 25 onwards?

Export the IP, MAC, and hostnames. :-)

nlbwmon seems to go into an infinite loop, using ~50% cpu, refusing to service the frontend. This happens after a few days, or sometimes a few weeks. I'm on MIPS, R6220. I see the same sort of strace as mentioned in a forum thread: lots of

recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base={{len=164, type=0x100 /* NLMSG_??? */, flags=0x600 /* NLM_F_??? */, seq=0, pid=0}, "\x02\x00\x00\x00\x3c\x00\x01\x80\x14\x00\x01\x80\x08\x00\x01\x00\xc0\xa8\x01\xf6\x08\x00\x02\x00\xc0\xa8\x01\x01\x24\x00\x02\x80"...}, iov_len=16384}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 164
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(7, {msg_namelen=12}, 0)         = -1 EAGAIN (Resource temporarily unavailable)

There are many more of the EAGAIN messages than the ones with more information. The command nlbw -c list just hangs:

root@OpenWrt:~# nlbw -c list

When build with the following stack protection
Following is encountered

Mon Aug  7 10:40:33 2017 kern.info kernel: [   45.628673] 
Mon Aug  7 10:40:33 2017 kern.info kernel: [   45.628673] do_page_fault(): sending SIGSEGV to nlbwmon for invalid write access to 00000000
Mon Aug  7 10:40:33 2017 kern.info kernel: [   45.637289] epc = 771a0c64 in libc.so[77180000+a0000]
Mon Aug  7 10:40:33 2017 kern.info kernel: [   45.642456] ra  = 564a91a8 in nlbwmon[564a4000+a000]
Mon Aug  7 10:40:33 2017 kern.info kernel: [   45.647550] 

  @@ -178,7 +178,7 @@ menu "Global build settings"
     	choice
     		prompt "User space Stack-Smashing Protection"
     		depends on USE_MUSL
    -		default PKG_CC_STACKPROTECTOR_REGULAR
    +		default PKG_CC_STACKPROTECTOR_STRONG
     		help
     		  Enable GCC Stack Smashing Protection (SSP) for userspace applications
     		config PKG_CC_STACKPROTECTOR_NONE
    @@ -196,7 +196,7 @@ menu "Global build settings"
     
     	choice
     		prompt "Kernel space Stack-Smashing Protection"
    -		default KERNEL_CC_STACKPROTECTOR_REGULAR
    +		default KERNEL_CC_STACKPROTECTOR_STRONG
     		depends on USE_MUSL || !(x86_64 || i386)
     		help
     		  Enable GCC Stack-Smashing Protection (SSP) for the kernel

Hi,

I compiled and deployed the nlbwmon package on my router running Openwrt chaos calmer (I created my own Makefile to be able to compile the package).
When I start it via cli, I get the following error:
nlbwmon
No interval specified; assuming 1st of month
Unable to read protocol list /usr/share/nlbwmon/protocols: No such file or directory
Apparently, I need to create a file with the list of protocols. Can you please specify the list of config options needed to have it working correctly? do I need to specify the database-folder?
How can I also use the client utility (nlbw) in order to query the daemon for current statistics?

Thank you

I run other linux/nonopenwrt dnsmasq servers and to keep it consisten
and ease of managment on openwrt side i use addn-hosts and dhcp-hostsfile
files for dnsmasq (instead of openwrt /etc/config/dhcp file).

addn-hosts file seems to be pickued up by nlbwmon for static ip to hostname lookups.
But not dhcp-hostsfile for mac to hostname (unless there is a current lease).

I have duplicated dhcp-hostsfile to openwrts /etc/config/dhcp config in the past
to get nlbwmon to lookup up the host names from mac addresses.

But would be nice if nlbwmon had an option to include dhcp-hostsfile
as an additional custom lookup locations/format.

dhcp-hostfile(for dhcp assignments) uses single line "mac,ip,hostname" format
ethers file can be used but only allows use of ip or hostname not both.

Also i use them because they can be reloaded after adding new hosts
without restarting dnsmasq, by just sending a HUP or reload to dnsmasq.
Unlike the standard openwrt /etc/config/dhcp config file which need full dnsmasq restart.

thank you

Not sure if this is relevant outside of my niche use case, but I spent several hours trying to debug why all my packet counters were zero and finally realized that I needed the following sysctl:

net.netfilter.nf_conntrack_acct = 1

This was set to 0 on my system. Setting it to 1 and restarting nlbwmon fixed it.
Might be helpful for someone in the future.

Dear @jow-

This is simply a question, and not an issue/bug. 😸

I am trying to verify nlbwmon's accuracy/comprehensiveness in monitoring bandwidth, and for my experiment, it would be handy if I could get a more granular view -- in other words, hourly breakdown. Is this possible? I'd love to see it in nlbwmon-luci, as I'm more comfortable with GUI, but will be okay if there is only a terminal/console way. I read the readme on https://github.com/jow-/nlbwmon, and I don't see terminal instructions.

Is it able to make hourly account interval?

I am suffering from some kind of peak hour issue.
However the current version cannot help me to track down the device even with 1 day interval now.

It would be helpful if I can have more smaller interval options.

Add similar to littlesnitch to show the region where the router initiated the link on the map

Currently nlbwmon fails to categorise all traffic correctly. I assumed at first that this was due to NATting, but now I'm no longer sure.

I can see in parse_proto_port you test the destination port against the values generated by the protocols file. In the case of an nlbwmon session running on an LEDE router (running NAT) an outgoing connection to (say) SSH is categorised correctly. An incoming connection that is DNATted via a forwarding rule to a session inside the internal network to SSH is categorised incorrectly as 'other'.

in nlbwmon setting in luci I can only see two options for local network.
one in called wan, which refers to my pppoe-wan interface and another is called lan that refers to my two wireless interfaces, on 2.4 and 5Gh.
but there is no sign of my LAN interfaces, eth0 that is for wan and eth1 which is for ethernet lan. and also no sign of the br-lan which is the bridge between two wirelesses and the ethernet lan (eth10).
is this correct?
shouldnt it show ethernet lan at least?

I noticed that my wireless connected devices traffic is not counted (almost at all).
if I update my app on my tablet for some 50MB traffic then the traffic on nlbwmon is not counted for that device at all.

also how can I set my option to only count my wan (internet traffic)
the router that lede is installed is connected in bridge mode to internet modem.(on eth0 port)

this is my ip command output on lede.
`

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 532
link/ether xxx.xxx.xxx.xxx brd ff:ff:ff:ff:ff:ff

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP group default qlen 532
link/etherxxx.xxx.xxx.xxx brd ff:ff:ff:ff:ff:ff

8: teql0: mtu 1500 qdisc noop state DOWN group default qlen 100
link/void

9: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether xxx.xxx.xxx.xxx brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
valid_lft forever preferred_lft forever

11: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP group default qlen 1000
link/ether xxx.xxx.xxx.xxx brd ff:ff:ff:ff:ff:ff

12: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP group default qlen 1000
link/ether xxx.xxx.xxx.xxx brd ff:ff:ff:ff:ff:ff

13: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc hfsc state UNKNOWN group default qlen 32
link/ether xxx.xxx.xxx.xxx brd ff:ff:ff:ff:ff:ff

19: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1430 qdisc hfsc state UNKNOWN group default qlen 3
link/ppp
inet xxx.xxx.xxx.xxx peer 10.255.255.255/32 scop
`