Running as a service problem about libreqos

[rchac]

Awesome! Nicely done

[rchac]

Try running sudo python3 ./LibreQoS.py then see what output shows. Theres probably something wrong at that step, preventing the actual creation of qdiscs for the ./graphBandwidth.py part to parse.

[jgamos]

this is the output

ubuntu@ubuntu:~/LibreQoS/v1.1$ sudo python3 ./LibreQoS.py
tc filter delete dev enp1s0f0
Error: Class doesn't support blocks.
We have an error talking to the kernel
tc filter delete dev enp1s0f0 root
Error: Specified class doesn't exist.
We have an error talking to the kernel
tc qdisc delete dev enp1s0f0 root
tc qdisc delete dev enp1s0f0
Error: Failed to find qdisc with specified handle.
tc filter delete dev enp1s0f1
Error: Class doesn't support blocks.
We have an error talking to the kernel
tc filter delete dev enp1s0f1 root
Error: Specified class doesn't exist.
We have an error talking to the kernel
tc qdisc delete dev enp1s0f1 root
tc qdisc delete dev enp1s0f1
Error: Failed to find qdisc with specified handle.
This Network Interface Card has 4 queues avaialble.
./xdp-cpumap-tc/bin/xps_setup.sh -d enp1s0f0 --default --disable
./xdp-cpumap-tc/bin/xps_setup.sh -d enp1s0f1 --default --disable
./xdp-cpumap-tc/src/xdp_iphash_to_cpu --dev enp1s0f0 --lan
libbpf: Error loading BTF: Invalid argument(22)
libbpf: magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 504
str_off: 504
str_len: 1732
btf_total_size: 2260
[1] PTR (anon) type_id=2
[2] STRUCT xdp_md size=24 vlen=6
data type_id=3 bits_offset=0
data_end type_id=3 bits_offset=32
data_meta type_id=3 bits_offset=64
ingress_ifindex type_id=3 bits_offset=96
rx_queue_index type_id=3 bits_offset=128
egress_ifindex type_id=3 bits_offset=160
[3] TYPEDEF __u32 type_id=4
[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] FUNC_PROTO (anon) return=6 args=(1 ctx)
[6] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[7] FUNC xdp_program type_id=5
[8] STRUCT bpf_map_def size=24 vlen=6
type type_id=4 bits_offset=0
key_size type_id=4 bits_offset=32
value_size type_id=4 bits_offset=64
max_entries type_id=4 bits_offset=96
map_flags type_id=4 bits_offset=128
inner_map_idx type_id=4 bits_offset=160
[9] VAR map_ip_hash type_id=8 linkage=1
[10] VAR map_txq_config type_id=8 linkage=1
[11] VAR map_ifindex_type type_id=8 linkage=1
[12] VAR cpu_map type_id=8 linkage=1
[13] VAR cpus_available type_id=8 linkage=1
[14] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[15] ARRAY (anon) type_id=14 index_type_id=16 nr_elems=4
[16] INT ARRAY_SIZE_TYPE size=4 bits_offset=0 nr_bits=32 encoding=(none)
[17] VAR _license type_id=15 linkage=1
[18] DATASEC license size=0 vlen=1 size == 0

libbpf: Error loading ELF section .BTF: -22. Ignored and continue.
TC workaround for mapname: map_ifindex_type map_fd:7

Added CPU:0 queue_size:2048

Added CPU:1 queue_size:2048

Added CPU:2 queue_size:2048

Added CPU:3 queue_size:2048

Documentation:

XDP: Lookup IPv4 and redirect to CPU hash

This program loads the XDP eBPF program into the kernel.

Use the cmdline tool for add/removing dest IPs to the hash

• Attached to device:enp1s0f0 (ifindex:3) prog_id:48

./xdp-cpumap-tc/src/xdp_iphash_to_cpu --dev enp1s0f1 --wan
libbpf: Error loading BTF: Invalid argument(22)
libbpf: magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 504
str_off: 504
str_len: 1732
btf_total_size: 2260
[1] PTR (anon) type_id=2
[2] STRUCT xdp_md size=24 vlen=6
data type_id=3 bits_offset=0
data_end type_id=3 bits_offset=32
data_meta type_id=3 bits_offset=64
ingress_ifindex type_id=3 bits_offset=96
rx_queue_index type_id=3 bits_offset=128
egress_ifindex type_id=3 bits_offset=160
[3] TYPEDEF __u32 type_id=4
[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] FUNC_PROTO (anon) return=6 args=(1 ctx)
[6] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[7] FUNC xdp_program type_id=5
[8] STRUCT bpf_map_def size=24 vlen=6
type type_id=4 bits_offset=0
key_size type_id=4 bits_offset=32
value_size type_id=4 bits_offset=64
max_entries type_id=4 bits_offset=96
map_flags type_id=4 bits_offset=128
inner_map_idx type_id=4 bits_offset=160
[9] VAR map_ip_hash type_id=8 linkage=1
[10] VAR map_txq_config type_id=8 linkage=1
[11] VAR map_ifindex_type type_id=8 linkage=1
[12] VAR cpu_map type_id=8 linkage=1
[13] VAR cpus_available type_id=8 linkage=1
[14] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[15] ARRAY (anon) type_id=14 index_type_id=16 nr_elems=4
[16] INT ARRAY_SIZE_TYPE size=4 bits_offset=0 nr_bits=32 encoding=(none)
[17] VAR _license type_id=15 linkage=1
[18] DATASEC license size=0 vlen=1 size == 0

libbpf: Error loading ELF section .BTF: -22. Ignored and continue.
TC workaround for mapname: map_ifindex_type map_fd:7

Added CPU:0 queue_size:2048

Added CPU:1 queue_size:2048

Added CPU:2 queue_size:2048

Added CPU:3 queue_size:2048

Documentation:

XDP: Lookup IPv4 and redirect to CPU hash

This program loads the XDP eBPF program into the kernel.

Use the cmdline tool for add/removing dest IPs to the hash

• Attached to device:enp1s0f1 (ifindex:4) prog_id:51

./xdp-cpumap-tc/src/xdp_iphash_to_cpu_cmdline --clear
iphash_modify() IP:10.10.10.6 key:0x60A0A0A TC-handle:0x0
key: 0x60A0A0A

./xdp-cpumap-tc/src/tc_classify --dev-egress enp1s0f0
TC: Control program for tc_classify_kern.o

• When using --dev, loads TC-egress filter calling BPF program

• Config of map_txq_config, that control CPU to queue_mapping

• List current queue_mapping (txq) config via --list

Map filename: /sys/fs/bpf/tc/globals/map_txq_config

Dev:enp1s0f0 -- Loading: TC-clsact egress

• Run: tc qdisc del dev enp1s0f0 clsact 2> /dev/null

• Run: tc qdisc add dev enp1s0f0 clsact

• Run: tc filter add dev enp1s0f0 egress prio 1 handle 1 bpf da obj ./xdp-cpumap-tc/src/tc_classify_kern.o sec tc_classify

./xdp-cpumap-tc/src/tc_classify --dev-egress enp1s0f1
TC: Control program for tc_classify_kern.o

• When using --dev, loads TC-egress filter calling BPF program

• Config of map_txq_config, that control CPU to queue_mapping

• List current queue_mapping (txq) config via --list

Map filename: /sys/fs/bpf/tc/globals/map_txq_config

Dev:enp1s0f1 -- Loading: TC-clsact egress

• Run: tc qdisc del dev enp1s0f1 clsact 2> /dev/null

• Run: tc qdisc add dev enp1s0f1 clsact

• Run: tc filter add dev enp1s0f1 egress prio 1 handle 1 bpf da obj ./xdp-cpumap-tc/src/tc_classify_kern.o sec tc_classify

tc qdisc replace dev enp1s0f0 root handle 7FFF: mq
tc qdisc add dev enp1s0f0 parent 7FFF:1 handle 1: htb default 2
tc class add dev enp1s0f0 parent 1: classid 1:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f0 parent 1:1 cake diffserv4
tc class add dev enp1s0f0 parent 1:1 classid 1:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f0 parent 1:2 cake diffserv4
tc qdisc add dev enp1s0f0 parent 7FFF:2 handle 2: htb default 2
tc class add dev enp1s0f0 parent 2: classid 2:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f0 parent 2:1 cake diffserv4
tc class add dev enp1s0f0 parent 2:1 classid 2:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f0 parent 2:2 cake diffserv4
tc qdisc add dev enp1s0f0 parent 7FFF:3 handle 3: htb default 2
tc class add dev enp1s0f0 parent 3: classid 3:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f0 parent 3:1 cake diffserv4
tc class add dev enp1s0f0 parent 3:1 classid 3:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f0 parent 3:2 cake diffserv4
tc qdisc add dev enp1s0f0 parent 7FFF:4 handle 4: htb default 2
tc class add dev enp1s0f0 parent 4: classid 4:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f0 parent 4:1 cake diffserv4
tc class add dev enp1s0f0 parent 4:1 classid 4:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f0 parent 4:2 cake diffserv4
tc qdisc replace dev enp1s0f1 root handle 7FFF: mq
tc qdisc add dev enp1s0f1 parent 7FFF:1 handle 1: htb default 2
tc class add dev enp1s0f1 parent 1: classid 1:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f1 parent 1:1 cake diffserv4
tc class add dev enp1s0f1 parent 1:1 classid 1:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f1 parent 1:2 cake diffserv4
tc qdisc add dev enp1s0f1 parent 7FFF:2 handle 2: htb default 2
tc class add dev enp1s0f1 parent 2: classid 2:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f1 parent 2:1 cake diffserv4
tc class add dev enp1s0f1 parent 2:1 classid 2:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f1 parent 2:2 cake diffserv4
tc qdisc add dev enp1s0f1 parent 7FFF:3 handle 3: htb default 2
tc class add dev enp1s0f1 parent 3: classid 3:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f1 parent 3:1 cake diffserv4
tc class add dev enp1s0f1 parent 3:1 classid 3:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f1 parent 3:2 cake diffserv4
tc qdisc add dev enp1s0f1 parent 7FFF:4 handle 4: htb default 2
tc class add dev enp1s0f1 parent 4: classid 4:1 htb rate 800mbit ceil 800mbit
tc qdisc add dev enp1s0f1 parent 4:1 cake diffserv4
tc class add dev enp1s0f1 parent 4:1 classid 4:2 htb rate 75.0mbit ceil 300mbit prio 5
tc qdisc add dev enp1s0f1 parent 4:2 cake diffserv4

Site_1
Download: 760 to 800 Mbps
Upload: 760 to 800 Mbps
tc class add dev enp1s0f0 parent 1:1 classid 3 htb rate 760mbit ceil 800mbit prio 3
tc class add dev enp1s0f1 parent 1:1 classid 3 htb rate 760mbit ceil 800mbit prio 3

RB1
Download: 55 to 545 Mbps
Upload: 22 to 218 Mbps
tc class add dev enp1s0f0 parent 1:3 classid 4 htb rate 55mbit ceil 545mbit prio 3
tc qdisc add dev enp1s0f0 parent 1:4 cake diffserv4
tc class add dev enp1s0f1 parent 1:3 classid 4 htb rate 22mbit ceil 218mbit prio 3
tc qdisc add dev enp1s0f1 parent 1:4 cake diffserv4
./xdp-cpumap-tc/src/xdp_iphash_to_cpu_cmdline --add --ip 10.10.10.6 --cpu 0 --classid 1:4
iphash_modify() IP:10.10.10.6 key:0x60A0A0A TC-handle:0x10004
key: 0x60A0A0A

Successful run completed on 29/06/2022 15:28:08
Program complete
ubuntu@ubuntu:~/LibreQoS/v1.1$

• I have this error at the beginning related to qdisc but when I tested the network with shaper it works just fine and latency is good while under heavy load.

[rchac]

What is the output of tc -s qdisc show dev enp1s0f1 ?

[jgamos]

here is the output

ubuntu@ubuntu:~/LibreQoS/v1.1$ tc -s qdisc show dev enp1s0f1
qdisc mq 7fff: root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 2: parent 7fff:2 r2q 10 default 0x2 direct_packets_stat 0 direct_qlen 1000
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 4: parent 7fff:4 r2q 10 default 0x2 direct_packets_stat 0 direct_qlen 1000
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: parent 7fff:1 r2q 10 default 0x2 direct_packets_stat 0 direct_qlen 1000
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 3: parent 7fff:3 r2q 10 default 0x2 direct_packets_stat 0 direct_qlen 1000
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 800a: parent 1:2 bandwidth unlimited diffserv4 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 65535 / 0
min/max overhead-adjusted size: 65535 / 0
average network hdr offset: 0

               Bulk  Best Effort        Video        Voice

thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 0us 0us 0us
av_delay 0us 0us 0us 0us
sp_delay 0us 0us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 0 0 0
bytes 0 0 0 0
way_inds 0 0 0 0
way_miss 0 0 0 0
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 0 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 0 0 0
quantum 1514 1514 1514 1514

qdisc cake 800c: parent 2:2 bandwidth unlimited diffserv4 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 65535 / 0
min/max overhead-adjusted size: 65535 / 0
average network hdr offset: 0

               Bulk  Best Effort        Video        Voice

thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 0us 0us 0us
av_delay 0us 0us 0us 0us
sp_delay 0us 0us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 0 0 0
bytes 0 0 0 0
way_inds 0 0 0 0
way_miss 0 0 0 0
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 0 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 0 0 0
quantum 1514 1514 1514 1514

qdisc cake 800e: parent 3:2 bandwidth unlimited diffserv4 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 65535 / 0
min/max overhead-adjusted size: 65535 / 0
average network hdr offset: 0

               Bulk  Best Effort        Video        Voice

thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 0us 0us 0us
av_delay 0us 0us 0us 0us
sp_delay 0us 0us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 0 0 0
bytes 0 0 0 0
way_inds 0 0 0 0
way_miss 0 0 0 0
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 0 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 0 0 0
quantum 1514 1514 1514 1514

qdisc cake 8010: parent 4:2 bandwidth unlimited diffserv4 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 65535 / 0
min/max overhead-adjusted size: 65535 / 0
average network hdr offset: 0

               Bulk  Best Effort        Video        Voice

thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 0us 0us 0us
av_delay 0us 0us 0us 0us
sp_delay 0us 0us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 0 0 0
bytes 0 0 0 0
way_inds 0 0 0 0
way_miss 0 0 0 0
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 0 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 0 0 0
quantum 1514 1514 1514 1514

qdisc cake 8012: parent 1:4 bandwidth unlimited diffserv4 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 0b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 65535 / 0
min/max overhead-adjusted size: 65535 / 0
average network hdr offset: 0

               Bulk  Best Effort        Video        Voice

thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 0us 0us 0us
av_delay 0us 0us 0us 0us
sp_delay 0us 0us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 0 0 0
bytes 0 0 0 0
way_inds 0 0 0 0
way_miss 0 0 0 0
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 0 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 0 0 0
quantum 1514 1514 1514 1514

qdisc clsact ffff: parent ffff:fff1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
ubuntu@ubuntu:~/LibreQoS/v1.1$

[jgamos]

additional info show output

ubuntu@ubuntu:/LibreQoS/v1.1$ sudo dpkg -s linux-modules-extra-$(uname -r) | grep Status
Status: install ok installed
ubuntu@ubuntu:/LibreQoS/v1.1$ sudo dpkg -s linux-image-generic | grep Status
Status: install ok installed
ubuntu@ubuntu:/LibreQoS/v1.1$ uname -r
5.13.0-52-generic
ubuntu@ubuntu:/LibreQoS/v1.1$

[rchac]

No data has moved through the qdiscs on enp1s0f1 according to these stats. graphBandwidth cannot parse all zero data.

[rchac]

Also graphLatency.py takes 20-30 seconds to execute. That may be why it appears "stuck".

[jgamos]

I did the instructions on "Running as a service" part then run "sudo systemctl start LibreQoS.service" but after "sudo systemctl status LibreQoS.service" i get the "Failed to start LibreQoS.service."

[rchac]

Please make sure in the file /etc/systemd/system/LibreQoS.service that you replaced $USER with the actual username - which in your case appears to be ubuntu. If it's left as $USER you'll get file path issues. Once that is fixed, try sudo systemctl restart LibreQoS.service. If it fails, run cat /var/log/syslog and see what you can find to troubleshoot the cause of the failure.

[jgamos]

it's fixed and actively running now as it should be, overlooked the next line as my editor highlighted the "$HOME" in dark blue color. (used Putty)

Thank you very much @rchac