Dynamic entries in Shaper.csv about libreqos

[marsalans]

I didn't expect this fast reply 😉 i'll try this in morning 🌄. I have a question related to bridge if it can be asked here or should i make another ticket for it. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, April 25, 2022 3:16:28 AM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Feel free to test it out. https://github.com/rchac/LibreQoS/blob/main/v1.1/lqAPI.py Although it's intended for v1.1, it will probably work for other versions. First just add these variables to ispConfig.py: # API Auth apiUsername = "testUser" apiPassword = "changeme8343486806" apiHostIP = "127.0.0.1" apiHostPost = 5000 To start API server: python3 lqAPI.py You can use Hoppscotch<https://hoppscotch.io/> to test API. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA5HVDUQRFT4NJJRFLLVGXB3ZANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[marsalans]

Thank you, you explain it very well. 🙂 Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, April 25, 2022 5:21:56 AM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) the bridge is transparent to customer router. both interface are in bridge mode and interface facing through wan and router are in same bridge. Pretty much this. The bridge is transparent. LibreQoS has no actual routing happening. Traffic flows thorough the LibreQoS bridge while its traveling between the edge router and core router on your network. The edge router and core router do not "see" the LibreQoS middle-box from a routing perspective. Whether you use Preseem, Paraqum, or LibreQoS, this sort of separation of functions is required. Most small ISPs start out with just one main router acting as an edge and core, where basically one big router does NAT, BGP, queues, PPPoE, etc. The first step to using a middle-box such as LibreQoS or Preseem is to separate the functions of your primary router between Edge and Core functions. This way, your new Core (Distribution) router will allow you to connect as many routers from different sites as you need to, without having to turn the LibreQoS box into a router itself. This keeps things much more manageable, and allows more traffic to flow through. one more thing that if we want to add multiple router to a single libreqos what will be the bridge configuration as other routers do not connect physically but logically through ip address and physical interfaces are just two for it. Once you split the functions of the edge router to Edge Router + Core Router, you can connect as many sites as you need to, to the Core router. For example, lets say your network uses one RB4011 as your Edge router, and you don't have a Core router yet. No problem, set up a lab environment with 2 RB4011s. Copy the configuration from your production RB4011 to both of these lab routers. Now remove unnecessary functions from the Core Router and Edge Router in the lab, changing IP addresses as needed to avoid confusion. Once you have this working, introduce LibreQoS to intercept the data going from the Core Router to the Edge router over a low-cost OSPF path (cost of 10 for example). Make sure to have a second, backup path directly connecting the edge and core. This backup path can have a high OSPF cost (like 200 for example). That way if your LibreQoS server ever turns off or crashes, traffic can flow around that path through the direct, backup link. [Diagram]<https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/design.png?raw=true> — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA27GKGQRWXC32ON53DVGXQSJANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rchac]

Sounds good. Currently working on a REST API that lets you GET, POST, PUT, or DELETE for Shaper.csv. It will also allow you to refresh LibreQoS according the contents of the Shaper.csv using an API call. Will release as soon as I can.

[marsalans]

sounds interesting, i deliberately wait for it. 😍 Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Sunday, April 24, 2022 11:24:30 PM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Sounds good. Currently working on a REST API that lets you GET, POST, PUT, or DELETE for Shaper.csv. It will also allow you to refresh LibreQoS according the contents of the Shaper.csv using an API call. Will release as soon as I can. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA3FGOKZ5UANPFBEC5DVGWGV5ANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rchac]

Feel free to test it out. https://github.com/rchac/LibreQoS/blob/main/v1.1/lqAPI.py
Although it's intended for v1.1, it will probably work for other versions.
First just add these variables to ispConfig.py:

# API Auth
apiUsername = "testUser"
apiPassword = "changeme8343486806"
apiHostIP = "127.0.0.1"
apiHostPost = 5000

To start API server: python3 lqAPI.py
You can use Hoppscotch to test API.

[marsalans]

I've been following this project for a long time but due to static file entries i do not make progress in lab test. In configuration example, a bridge is created between two interfaces. what purpose it is serving ? like for a single in/out excluding management interface one wan and one lan, for a router libreqos box act as a wan ? assign gateway ip to libreqos on one interface and the other interface to customer router. or the bridge is transparent to customer router both interface are in bridge mode and interface facing through wan and router are in same bridge. one more thing that if we want to add multiple router to a single libreqos what will be the bridge configuration as other routers do not connect physically but logically through ip address and physical interfaces are just two for it. Thanks Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Muhammad Arsalan Siddiqui ***@***.***> Sent: Monday, April 25, 2022 3:52:23 AM To: rchac/LibreQoS ***@***.***>; rchac/LibreQoS ***@***.***> Cc: Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) I didn't expect this fast reply 😉 i'll try this in morning 🌄. I have a question related to bridge if it can be asked here or should i make another ticket for it. Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, April 25, 2022 3:16:28 AM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Feel free to test it out. https://github.com/rchac/LibreQoS/blob/main/v1.1/lqAPI.py Although it's intended for v1.1, it will probably work for other versions. First just add these variables to ispConfig.py: # API Auth apiUsername = "testUser" apiPassword = "changeme8343486806" apiHostIP = "127.0.0.1" apiHostPost = 5000 To start API server: python3 lqAPI.py You can use Hoppscotch<https://hoppscotch.io/> to test API. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA5HVDUQRFT4NJJRFLLVGXB3ZANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rchac]

the bridge is transparent to customer router. both interface are in bridge mode and interface facing through wan and router are in same bridge.

Pretty much this. The bridge is transparent. LibreQoS has no actual routing happening. Traffic flows thorough the LibreQoS bridge while its traveling between the edge router and core router on your network. The edge router and core router do not "see" the LibreQoS middle-box from a routing perspective. Whether you use Preseem, Paraqum, or LibreQoS, this sort of separation of functions is required.

Most small ISPs start out with just one main router acting as an edge and core, where basically one big router does NAT, BGP, queues, PPPoE, etc. The first step to using a middle-box such as LibreQoS or Preseem is to separate the functions of your primary router between Edge and Core functions. This way, your new Core (Distribution) router will allow you to connect as many routers from different sites as you need to, without having to turn the LibreQoS box into a router itself. This keeps things much more manageable, and allows more traffic to flow through.

one more thing that if we want to add multiple router to a single libreqos what will be the bridge configuration as other routers do not connect physically but logically through ip address and physical interfaces are just two for it.

Once you split the functions of the edge router to Edge Router + Core Router, you can connect as many sites as you need to, to the Core router. For example, lets say your network uses one RB4011 as your Edge router, and you don't have a Core router yet. No problem, set up a lab environment with 2 RB4011s. Copy the configuration from your production RB4011 to both of these lab routers. Now remove unnecessary functions from the Core Router and Edge Router in the lab, changing IP addresses as needed to avoid confusion. Once you have this working, introduce LibreQoS to intercept the data going from the Core Router to the Edge router over a low-cost OSPF path (cost of 10 for example). Make sure to have a second, backup path directly connecting the edge and core. This backup path can have a high OSPF cost (like 200 for example). That way if your LibreQoS server ever turns off or crashes, traffic can flow around that path through the direct, backup link.

[marsalans]

I have one more thing in mind, can we have dual queue for single IP ? in a sense that some services like facebook, youtube have cdn and ISP gives separate bandwidth for it. We make a separate ip list for it and then mark connection/packets for those IP and make pcc rules for the queue. Can we do something like this with libreqos ? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Muhammad Arsalan Siddiqui ***@***.***> Sent: Monday, April 25, 2022 5:27:20 AM To: rchac/LibreQoS ***@***.***>; rchac/LibreQoS ***@***.***> Cc: Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Thank you, you explain it very well. 🙂 Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, April 25, 2022 5:21:56 AM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) the bridge is transparent to customer router. both interface are in bridge mode and interface facing through wan and router are in same bridge. Pretty much this. The bridge is transparent. LibreQoS has no actual routing happening. Traffic flows thorough the LibreQoS bridge while its traveling between the edge router and core router on your network. The edge router and core router do not "see" the LibreQoS middle-box from a routing perspective. Whether you use Preseem, Paraqum, or LibreQoS, this sort of separation of functions is required. Most small ISPs start out with just one main router acting as an edge and core, where basically one big router does NAT, BGP, queues, PPPoE, etc. The first step to using a middle-box such as LibreQoS or Preseem is to separate the functions of your primary router between Edge and Core functions. This way, your new Core (Distribution) router will allow you to connect as many routers from different sites as you need to, without having to turn the LibreQoS box into a router itself. This keeps things much more manageable, and allows more traffic to flow through. one more thing that if we want to add multiple router to a single libreqos what will be the bridge configuration as other routers do not connect physically but logically through ip address and physical interfaces are just two for it. Once you split the functions of the edge router to Edge Router + Core Router, you can connect as many sites as you need to, to the Core router. For example, lets say your network uses one RB4011 as your Edge router, and you don't have a Core router yet. No problem, set up a lab environment with 2 RB4011s. Copy the configuration from your production RB4011 to both of these lab routers. Now remove unnecessary functions from the Core Router and Edge Router in the lab, changing IP addresses as needed to avoid confusion. Once you have this working, introduce LibreQoS to intercept the data going from the Core Router to the Edge router over a low-cost OSPF path (cost of 10 for example). Make sure to have a second, backup path directly connecting the edge and core. This backup path can have a high OSPF cost (like 200 for example). That way if your LibreQoS server ever turns off or crashes, traffic can flow around that path through the direct, backup link. [Diagram]<https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/design.png?raw=true> — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA27GKGQRWXC32ON53DVGXQSJANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rchac]

Interesting. If I'm understanding the scenario correctly, one way you could approach it would be to run two VMs of LibreQoS. You could run the two VMs on a single server with two dual Network Interface Cards, running proxmox as the hypervisor. The first LibreQoS VM would just have static rate limits for the IP addresses of the CDNs (say, 2Gbps for one CDN, 1Gbps for the other CDN). This instance would not need an API or statistics/graphing, so it would use minimal resources. Then, the second LibreQoS instance would do rate limits for the individual subscribers on your network, allowing them to get appropriate max bandwidth and still utilize multiple CDNs efficiently. That would keep bufferbloat and latency low compared to PCQ, and limit resource utilization of your routers.

[marsalans]

If i would create a separate vm of libreqos how would i make sure that only cdn ips will go through it as all the traffic starts flowing out from both vms and there is 99% chance that on the customer end speed is doubled. And according to you adding only static ip in second vm of libreqos as destination pass traffic without limitation and can be abused. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, April 25, 2022 6:12:42 AM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Interesting. If I'm understanding the scenario correctly, one way you could approach it would be to run two VMs of LibreQoS. You could run the two VMs on a single server with two dual Network Interface Cards, running proxmox as the hypervisor. The first LibreQoS VM would just have static rate limits for the IP addresses of the CDNs (say, 2Gbps for one CDN, 1Gbps for the other CDN). This instance would not need an API or statistics/graphing, so it would use minimal resources. Then, the second LibreQoS instance would do rate limits for the individual subscribers on your network, allowing them to get appropriate max bandwidth and still utilize multiple CDNs efficiently. That would keep bufferbloat and latency low compared to PCQ, and limit resource utilization of your routers. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA7I7L4TOZEBEBIHSMDVGXWQVANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rchac]

The traffic cannot leak out. As it passes through the bridge it has no conception that is going through a bridge because its a simple layer 2 tunnel separate from the management interface , so it has no way to take a different route.

LibreQoS VM1:
br0 = eth1 + eth2
eth0 = mgmt
Shapes: CDN IP addresses

LibreQoS VM2:
br0 = eth1 + eth2
eth0 = mgmt
Shapes: Your Customers' IP addresses

Here is an illustration for this scenario:

And according to you adding only static ip in second vm of libreqos as destination pass traffic without limitation and can be abused.

As long as one LibreQoS VM is shaping the CDN IPs, and the other LibreQoS VM is shaping the Customer IPs, you would not have any customers getting double bandwidth.

[marsalans]

I'm talking about shaping CDN IPs in a sense that, 1 client having service plan of 10 Mbps other have 50 Mbps and so on. By the first shaper which shapes only CDN traffic it doesn't know how to shape traffic by client service plan. As we are just giving 1 Gbps to each CDN IP. So few client with a service plan of 10 Mbps can fully saturate 1 Gbps unintentionally which is unfair for others and ISP. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, April 25, 2022 7:14:31 AM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) The traffic cannot leak out. As it passes through the bridge it has no conception that is going through a bridge because its a simple layer 2 tunnel separate from the management interface , so it has no way to take a different route. LibreQoS VM1: br0 = eth1 + eth2 eth0 = mgmt Shapes: CDN IP addresses LibreQoS VM2: br0 = eth1 + eth2 eth0 = mgmt Shapes: Your Customers' IP addresses Here is an illustration for this scenario: [Diagram]<https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/example238529.png?raw=true> And according to you adding only static ip in second vm of libreqos as destination pass traffic without limitation and can be abused. As long as one LibreQoS VM is shaping the CDN IPs, and the other LibreQoS VM is shaping the Customer IPs, you would not have any customers getting double bandwidth. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAAZMRIVUL4SLVVSVTA3VGX5YPANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rchac]

I re-read your original question and I think I probably misunderstood originally, my apologies. You wrote

in a sense that some services like facebook, youtube have cdn and ISP gives separate bandwidth for it. We make a separate ip list for it and then mark connection/packets for those IP and make pcc rules for the queue.

Could you help me better understand this? I am imagining two different things you might mean:

Is your goal to make sure each customers connection has equitable access to resources upstream? For example, so that a 10Mbps customer can still get a fair slice of Facebook, or YouTube, even though some customers may have higher bandwidth plans?

Or

Do you have a list of specific CDN IPs that you need to make sure do not exceed certain amounts of bandwidth. For example a Facebook CDN that you want to make sure does not exceed X Gbps. And you want each of your customers to get equitable access to these CDNs.

[marsalans]

No need to apologize indeed you have created a great project which no one thinks of.

Fair slice is only possible if ISP is not bottlenecking their bandwidth nor customer pipe is choked. It is also a great achievement and benefit for a customer and can be looked into future as i think preseem might have implement this.

But i'm talking about something else here,

This can be quoted with some modifications:

Do you have a list of specific CDN IPs that you need to make sure do not exceed certain amounts of bandwidth. For example a Facebook CDN that you want to make sure does not exceed X Gbps. And you want each of your customers to get equitable access to these CDNs.

What i'm saying that just like a normal service plan some upload/download speed is defined, and as a addon, CDN upload/download also needs to be defined which can be same as normal service plan up/down or can be greater. So someone can not saturate whole CDN pipe.

Now for the CDN solution as assumption, adding second box of libreqos and adding CDN IP means that a customer of 10 Mbps can download from anywhere with 10 Mbps but when they download from CDN they can get upto 1 Gbps if their connectivity is of Gigabit.

So the solution is to mark IP packets then use these packet marks in queue types. This solution works because we make a list of IPs which belong to which service plan and know the router knows which IP belongs to which customer and what cdn to be given to them.

The queues and marking i just told about is taken from Mikrotik RouterOS, because we use it and do all the stuff on same box.

Now my conclusion is if it is possible to limit up/down per IP in bridge so it is also possible to packet mark by IP address in bridge for cdn or other ISP hosted content.

[marsalans]

In the scenerion it would be great to add a queue seperately for source and destination ip address

[marsalans]

I was reading issues realted to this, and i found this #28
It is perfect for this task as we can create a list of IP here and assign queues with the ip list

example:

internet ip list for service plan 1
cdn ip list for service plan 1

internet ip list for service plan 2
cdn ip list for service plan 2

and so on, what do you think about it ?

[rchac]

I would recommend forking the project and attempting to develop that, if it would be helpful to your network. I unfortunately do not have the time or resources to develop that feature at the moment.

[marsalans]

I will sure work on it, @heistp @dtaht can i get some assistance in this ?

[heistp]

Hi- I don't have time for new development work, but just mentioning some tests of custom qdisc classification, in case that's useful somehow. I haven't read your plan fully to know if that's even what you need or want. Good luck...

[marsalans]

@heistp thanks for showing up, yeah i read the git and thats why i link that here and also this is what i wanted.

[dtaht]

I am not in a position to work for free at this point.

[marsalans]

I'm just requesting to guide me a little bit but i didnt meant that you have to do this for me.

[marsalans]

hey there, running lqAPI.py gives 404 error.

[rchac]

Use https://hoppscotch.io/ to test the api. The API server is running. The resources are /devices (adding or removing entries) and /shaper (restarting the shaper). To refresh , it's something like:
192.168.0.164:5000/shaper?refresh=True

For /devices, the parameters are: id, mac, parentNode, hostname, ipv4, ipv6, downloadMin, uploadMin, downloadMax, uploadMax

[marsalans]

Thanks it works,

I have developed this in python using api to get all the connected clients and there queues values are printed in a table.

As i use this on same machine as libreqos so should i use api or write directly to csv file and call refresh shaper ?
or to use api for csv writing.

I there are duplicate entries will api update to the last record or add another record ?

[rchac]

Nice! If you're already running it on the same machine, either approach is fine. I think writing to the CSV directly is a bit easier for integrations, for me personally. But both are fine. And if an entry already exists, and you try to submit it by the API, the API will return the error "[device] already exists".

[marsalans]

Thanks. I want that if an entry already exists it should replace itself with new banwidth limit as, i'm running pppoe and there are lots of connections/disconnections and their IP is not static. In this scenario if other client gets same IP after first client disconnected, other client queue does not change to reflect their service plan. I dont think this is a big issue, i can modify your code or write a logic if i will be writing directly to csv. Major issue bothering me is accurately setting site and ap bandwidth. Suppose i have 10 Gigabit pipe and i want to divide between sites, what should i do to balance these as per site and cpu restriction that 4 Gbps is the limit. As documentation says i have to go deeper and make parent sites how that will load balance between cores and the limit to pass 10 Gbps ? Setting this is my headache, as i dont have physical sites or AP just one or two router boards for single shaper. should i create 5 sites of 2 Gbps and randomly assign ip address to them or just 3 sites of 3300 Mbps ?

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, August 15, 2022 11:31:20 PM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Author ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Nice! If you're already running it on the same machine, either approach is fine. I think writing to the CSV directly is a bit easier for integrations, for me personally. But both are fine. And if an entry already exists, and you try to submit it by the API, the API will return the error "[device] already exists". — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA6URQ4I4LB4SESMA3DVZKEHRANCNFSM5TNVFFKQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dtaht]

Awesome. The feature that I deeply desire is to be able to do the equivalent of "tc -s qdisc show" on the AP or customer name, to pull out the up and/or down statistics.

Inventing a name for this rather than "app2", call it lqos...

lqos show htb cust AA (show the htb bins for up and down)

or

lqos -s show cust AA ()

lqos -s -j show ap AB (for json)

I use the "watch" facility a lot elsewhere...

watch tc -s qdisc show dev eth1 handle XX:YY

[rchac]

@marsalans

I would probably do 5 sites, just to balance out CPU load and for a safe margin - in case one CPU gets more small packets than is common.

Achieving a hard ceiling over multiple cores is super challenging because tc-HTB isn't multi-CPU yet, so we have do a workaround using xdp-cpumap-tc. None of the top-level nodes (v1.1) or sites (v1.0) are aware of each-others upper limit, so global limiting isn't really possible until some better HTB implementation is merged into the Linux kernel. However, your solution of assigning 2Gbps to 5 cores would likely solve this for your use case. =) I like that approach.

I am planning to add a feature where any devices in Shaper.csv without a listed Parent Node get assigned to CPU cores equitably, without having to make arbitrary parent nodes or sites. It would not be able to enforce a hard ceiling (such as 10Gbps at the edge) but would at least equally distribute across CPUs automatically.

@dtaht

Do you want lqos to be able to accept start/stop time and report total dl/ul MB for that period? I'm thinking we could have it poll InfluxDB and pull that info.

[marsalans]

Yes i first think of 5 sites then thought what happen if one site get more queues then other, so i re think about it and got 3.3 Gbps which is less then 4 Gbps and it will not overload a site and balancing is not required or will not create a problem. Same is the case with 4 site of 2.5 Gbps.

But as the concerned is CPU so 5 site is ideal and i have to work on balancing each site.

I hope you heard of hqos it utilize DiffServ model and can be used for multiple session and distinguish user-specific or service-specific traffic and provide differentiated bandwidth management.

Why i'm digging LibreQoS because i've seen a bng that serve pppoe clients and i'm amazed by its performance it uses vpp and hqos and i want to built a similar software. So beside LibreQos i'm also studying vpp, hqos, dpdk.

I also ask you about a solution where we can do multi queus with IP source/destination that solution is HQoS, at that time i didnt know that.

I search further and found that DPDK latest verion dont support HQoS.

I am planning to add a feature where any devices in Shaper.csv without a listed Parent Node get assigned to CPU cores equitably, without having to make arbitrary parent nodes or sites. It would not be able to enforce a hard ceiling (such as 10Gbps at the edge) but would at least equally distribute across CPUs automatically.

This would be great. A option which when enabled automatically balance the sites which are listed is csv would also be great.

I also wanted to share with you that the script i wrote to connect to mikrotik pppoe server takes 1.5 to 2 minutes to get all pppoe connected session along with their queue dl/ul. Currently i fetched details from our remote mikrotik over public WAN and there are 1800 entries there. My script is incomplete and i'm working on it, Alongside fetching details the script will then remove queues from mikrotik sequentially to relaase burden on it and let LibreQos do the hard job. Removing the queues add some delay and max it would take 4 minutes to do that.

I will soon share the tests result with you. I also want to know what are your future thought about this project ?

[marsalans]

And what role IRQbalance plays here ?

[rchac]

Yes i first think of 5 sites then thought what happen if one site get more queues then other, so i re think about it and got 3.3 Gbps which is less then 4 Gbps and it will not overload a site and balancing is not required or will not create a problem. Same is the case with 4 site of 2.5 Gbps.
But as the concerned is CPU so 5 site is ideal and i have to work on balancing each site.

Ok gotcha. Yeah I'm trying to implement binpacking to sort top-level parent nodes (including some generated to just to hold Shaper.csv entries without any parent node defined). That will hopefully automate all this.

I hope you heard of hqos it utilize DiffServ model and can be used for multiple session and distinguish user-specific or service-specific traffic and provide differentiated bandwidth management.

That's very cool. I had never seen this before. So it's part of VPP?

Why i'm digging LibreQoS because i've seen a bng that serve pppoe clients and i'm amazed by its performance it uses vpp and hqos and i want to built a similar software. So beside LibreQos i'm also studying vpp, hqos, dpdk.

Check out hardware-offload HTB. It's already usable for Mellanox cards. The drawback is that you can only go 3 levels deep in an HTB, and you're vendor locked to Mellanox, but supposedly it allows up to 100 Gbps or something crazy like that. For your use case that might be super efficient and practical vs having to create something with DPDK or VPP.

I also ask you about a solution where we can do multi queus with IP source/destination that solution is HQoS, at that time i didnt know that.

I had no idea about HQoS. It seems promising.

I search further and found that DPDK latest verion dont support HQoS.

Is it deprecated? Oh no. :/

This would be great. A option which when enabled automatically balance the sites which are listed is csv would also be great.

Working on it! The code may take me a bit because i'm super busy with the WISP operation but hopefully things calm down soon.

I also wanted to share with you that the script i wrote to connect to mikrotik pppoe server takes 1.5 to 2 minutes to get all pppoe connected session along with their queue dl/ul. Currently i fetched details from our remote mikrotik over public WAN and there are 1800 entries there. My script is incomplete and i'm working on it, Alongside fetching details the script will then remove queues from mikrotik sequentially to relaase burden on it and let LibreQos do the hard job. Removing the queues add some delay and max it would take 4 minutes to do that.
I will soon share the tests result with you. I also want to know what are your future thought about this project ?

That's great, awesome work. I like that approach. From what you're seeing, how does LibreQoS compare to just queues on the MikroTik?

[marsalans]

And what role IRQbalance plays here ?

Hi @rchac, can you tell me about this.

[rchac]

And what role IRQbalance plays here ?

Hi @rchac, can you tell me about this.

I am just learning about this. So apparently IRQbalance should be disabled when using XDP? https://suricata.readthedocs.io/en/suricata-5.0.10/capture-hardware/ebpf-xdp.html

[marsalans]

I think it will be required to disable as it does the same thing by balancing interrupts among cpu cores. can we select custom cpu cores for data plane ? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, August 29, 2022 7:47:39 PM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Mention ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) And what role IRQbalance plays here ? Hi @rchac<https://github.com/rchac>, can you tell me about this. I am just learning about this. So apparently IRQbalance should be disabled when using XDP? https://suricata.readthedocs.io/en/suricata-5.0.10/capture-hardware/ebpf-xdp.html — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA6OFAYFVP3QZLRLIS3V3TEQXANCNFSM5TNVFFKQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rchac]

Ok. I will test overnight with IRQbalance disabled on ubuntu. If no issue emerge I will add to the Wiki to disable IRQbalance as one of the steps.

And sort-of. Right now any top-level parent node in LibreQoS is assigned sequentially to each CPU core. So if you had 4 top-level parent nodes, they'd occupy the first 4 CPU cores. Once the number of top-level parent nodes hits the limit of CPU cores, it wraps back around to the first core.

[marsalans]

Okay, and what if there are more than four cores ? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, August 29, 2022, 19:55 To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Mention ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) Ok. I will test overnight with IRQbalance disabled on ubuntu. If no issue emerge I will add to the Wiki to disable IRQbalance as one of the steps. And sort-of. Right now any top-level parent node in LibreQoS is assigned sequentially to each CPU core. So if you had 4 top-level parent nodes, they'd occupy the first 4 CPU cores. Once the number of top-level parent nodes hits the limit of CPU cores, it wraps back around to the first core. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA7ETHOIFAQLB2MTYTDV3TFOPANCNFSM5TNVFFKQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rchac]

So lets say you have 12 cores.

If you have 12 cores, and 5 top-level parent nodes, all 5 top-level parent nodes are using one CPU core each, but the last 7 CPU cores are basically unused.

If you have 12 cores, and 12 top-level parent nodes, all top-level parent nodes are using one CPU core (super efficient!).

If you have 12 cores and 18 top-level parent nodes, it wraps around. So the first 6 cores hold two top-level parent nodes, and the last 6 cores have just one top-level parent nodes each (less ideal, but works).

The goal is to have one top-level parent node per CPU core, or a multiple, like two top-level parent nodes per CPU core.

[marsalans]

Parent node means site right ? and their sub node share with all cpu ? It is generally seen that low cpu count with high frequency do better than high number of cpu count with low frequency, Is this theory applied here ? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Robert Chacón ***@***.***> Sent: Monday, August 29, 2022 8:04:18 PM To: rchac/LibreQoS ***@***.***> Cc: Muhammad Arsalan Siddiqui ***@***.***>; Mention ***@***.***> Subject: Re: [rchac/LibreQoS] Dynamic entries in Shaper.csv (Issue #27) So lets say you have 12 cores. If you have 12 cores, and 5 top-level parent nodes, all 5 top-level parent nodes are using one CPU core each, but the last 7 CPU cores are basically unused. If you have 12 cores, and 12 top-level parent nodes, all top-level parent nodes are using one CPU core (super efficient!). If you have 12 cores and 18 top-level parent nodes, it wraps around. So the first 6 cores hold two top-level parent nodes, and the last 6 cores have just one top-level parent nodes each (less ideal, but works). The goal is to have one top-level parent node per CPU core, or a multiple, like two top-level parent nodes per CPU core. — Reply to this email directly, view it on GitHub<#27 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIYPAA5IVI7EBA5NL4X6X2TV3TGPFANCNFSM5TNVFFKQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rchac]

Parent node means site right ? and their sub node share with all cpu ?

Yes, for v1.0 and prior that would be the Site. All sub nodes share the same CPU.

It is generally seen that low cpu count with high frequency do better than high number of cpu count with low frequency, Is this theory applied here ?

In general, yes. But it depends on how much traffic you're moving and how small the packets are. Although you can put about 4Gbps through one CPU core, that assumes standard 1500 byte packet sizes, and a small number of CAKE instances. To account for the real world where many packets are smaller than that, I would recommend to use many cores as you need for your traffic load (maybe one core for every 2Gbps of throughput) and balance out processing among those cores.