Support for one-network-card card configuration, single card configuration, one NIC only about libreqos

[thebracket]

Is there a way to unlink this from issue 26? They aren't really related. 26 is talking about VLANs inside the bridge - which works really well, I'm pushing 2gbit/s through VLANs in a shaping bridge right now. This is talking about the top-level bridge members being VLANs. (We should probably close 26)

[thebracket]

Based on testing on the Equinix servers, the performance hit is pretty tiny. We've pushed 25 gbps through a single-card ("on a stick") setup on a 16 core (32 virtual cores, but we turned off hyperthreading) with CPU to spare. I honestly don't know what the upper limit is, since we're saturating our 25 gbps test and don't have anything bigger to test with!

…

On Thu, Jan 5, 2023 at 5:08 AM Interduo ***@***.***> wrote: I saw be2e2fb <be2e2fb> @thebracket <https://github.com/thebracket> do You know how big is performance penalty (and if is any) using only one interface instead of two? — Reply to this email directly, view it on GitHub <#48 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRU435CIRNPDPRC2VJRXY3WQ2TSDANCNFSM576E7CHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[thebracket]

I did a little digging (ran into some hyper-v issues with tagged VLANs between VMs; I need to get a better test environment locally!). So there's good news and bad news:

The good news is that binding XDP programs to VLAN interfaces works. I can bind an XDP program to a VLAN I've created (in this case eth1.8), and it functions.

The bad news is that performance is going to be really, really bad:

• There's only 1 queue on a VLAN interface. ls /sys/class/net/eth1.8/queues/ shows rx-0 and tx-0. So CPU mapping won't do anything, because there's only one queue to map - it's going to sit on a single core, no matter what you do.
• VLAN interfaces appear as "XDP generic" (which means "SKB bind mode" internally); if your interface had any hardware acceleration for XDP, it just went away.

So my advice: don't do this.

[interduo]

Sorry for not editing my earlier full post. I did it now - interfaces are named corectly in ispConfig.py and in netplan YAML. I did use queuesAvailableOverride=16.

That is really bad news like You said.

Maybe there is another (not vlans) idea/technology/design model for allowing us to use just one link?
Thinking...maybe we could tag inbound/outbound network packets by some mark/tag on Edge router and then change the tag on LibreQoS after queueing?

[interduo]

I would like to remove all SPOF's ... so get something like that:
https://raw.githubusercontent.com/interduo/LibreQoS/354141a766e19c313301b3f0824cc010330cc650/diagram-test.drawio

[interduo]

@rchac how big is performance dropdown when using virtio instead of passthrough NICs? Did You try to use SRIOV instead of PCI-Passthrough?

[rchac]

@rchac how big is performance dropdown when using virtio instead of passthrough NICs? Did You try to use SRIOV instead of PCI-Passthrough?

Virtio / generic XDP is about 1/4 the performance of pass-through NIC. Iam testing out a bare metal server and it's working great so far.

[interduo]

Did You tested SRIOV?

[rchac]

Did You tested SRIOV?

I don't think SRIOV works with XDP. But it may be possible.

[interduo]

If yes then we could make a LibreQoS OSPF neighbour and add two /32 networks between EDGE router and CORE switch on virtualized NICs. SRIOV overhead is about 2 or less %.

[dtaht]

We haven't been in a position to document or discuss virtualization all that much, and it makes my head hurt. We did just aquire some serious resources from equinix #151 to be able to test this more fully, but I kind of consider virtualization testing to be a whole other "project", after we get a solid test suite in #153. @interduo Interduo, though, since you have the resources, goferit with v1.3 and get back to us?

Secondly, I like the idea1 card operation a LOT, but don't know if that works in 1.3?

[interduo]

Now I use 1.2 version as VM. It's doing it works.

I will get You a feedback but my local roadmap needs few hw changes so it will take us few weeks. And I will test firstly one interface configuration.

[interduo]

I am trying to test this config:

My netplan is:

root@rtx-libreqos-new:~/LibreQoS/v1.3# cat /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
  ethernets:
    ens18:
      addresses:
      - 10.100.0.6/24
      routes:
      - to: default
        via: 10.100.0.254
      nameservers:
        addresses:
        - 46.151.191.151
        - 46.151.191.5
        search: []
    ens16np0:
      dhcp4: no
  version: 2
  vlans:
      v800@ens16np0:
          id: 800
          link: ens16np0
      v900@ens16np0:
          id: 900
          link: ens16np0
  bridges:
    br0:
      interfaces:
        - v800@ens16np0
        - v900@ens16np0
  version: 2

root@rtx-libreqos-new:~/LibreQoS/v1.3# cat ispConfig.py | grep interface

interfaceA = 'v800@ens16np0'
interfaceB = 'v900@ens16np0'

root@rtx-libreqos-new:~/LibreQoS/v1.3# ip a s | grep @ens16np0
5: v900@ens16np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
6: v800@ens16np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000

I got an error:

root@rtx-libreqos-new:~/LibreQoS/v1.3# ./LibreQoS.py 
refreshShapers starting at 23/11/2022 10:11:26
First time run since system boot.
Validating input files 'ShapedDevices.csv' and 'network.json'
network.json passed validation
ShapedDevices.csv passed validation
Backed up good config as lastGoodConfig.csv and lastGoodConfig.json
Traceback (most recent call last):
  File "/root/LibreQoS/v1.3/./LibreQoS.py", line 1216, in <module>
    refreshShapers()
  File "/root/LibreQoS/v1.3/./LibreQoS.py", line 442, in refreshShapers
    queuesAvailable = findQueuesAvailable()
  File "/root/LibreQoS/v1.3/./LibreQoS.py", line 94, in findQueuesAvailable
    directory_contents = os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/v800@ens16np0/queues/'

Whereas:
root@rtx-libreqos-new:~/LibreQoS/v1.3# ethtool -l ens16np0

gives:

Channel parameters for ens16np0:
Pre-set maximums:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	16
Current hardware settings:
RX:		n/a
TX:		n/a
Other:		n/a
Combined:	16

If VLAN is used, the LibreQoS should try find queues count in system using native interface.

root@rtx-libreqos-new:~/LibreQoS/v1.3# brctl show

bridge name	bridge id		STP enabled	interfaces
br0		8000.7e4911d668ca	no		v800@ens16np0
							v900@ens16np0

If I set:
cat /root/LibreQoS/v1.3/ispConfig.py | grep Over
queuesAvailableOverride = 16

It's got no error - but:

root@rtx-libreqos-new:~/LibreQoS/v1.3# tc class show dev v800@ens16np0
root@rtx-libreqos-new:~/LibreQoS/v1.3# tc class show dev v900@ens16np0

gives an empty output.

Full running log: http://kłopotek.pl/running_libre.txt

[rchac]

I noticed "LOWERLAYERDOWN" on your ip output. Is the physical interface not properly connected?

[thebracket]

Just to expand in case in anyone wants to test.

(as root)

1. Enable VLAN tagging support in your kernel with modprobe 8021q
2. Add a VLAN with ip link add link eth1 eth1.8 type vlan id 8
3. Turn the VLAN interface on with ip link set dev eth1.8 up

Now have a look at sys/class/net/eth1.8/queues and you should see 1 queue for each direction. 1 queue will work, and CPU map can still spray IP matches into different cores - but with only 1 in and 1 out, performance is going to be very limited at the interrupt level.

If you see more than 1 queue in each direction, you obviously have a nicer NIC than me and I'm thrilled to be proven wrong.

Using my test XDP program, the packets I see are un-tagged on that interface (as they should be), and I didn't see signs of leakage from the parent VLAN. So isolation is working.

[interduo]

I noticed "LOWERLAYERDOWN" on your ip output. Is the physical interface not properly connected?

Yes its ok - because its test instance. Switch port is administrativly down.

[interduo]

Vlan interface use NIC queues.

[thebracket]

Looking at the config, I see a couple of things:

interfaceA = 'v800@ens16np0'
interfaceB = 'v900@ens16np0'

Isn't going to work, because the devices are named v800 and v900. You should see entries in /sys/class/net/v800/queues and /sys/class/v900/queues.

Since there's almost certainly only 1 queue (per direction), you're going to want to look in /sys/class/net/ens12np0/queues and see how many there actually are. Add queuesAvailableOverride=(the number of queues) to your config to bypass the test.

[interduo]

So having two core switches and no SPOF comes with requirement now:
LibreQoS - 2
Second LibreQoS - 2
BGP - 1
Second BGP - 1
Internet Upsteams - >=2
another core switch - 2 (one switch-switch link)

So we need minimum 10x QSFP+ ports WRRRRR

That is not a thing for 10G network through but the stairs shows when we are above that line.
Just look at the market and see that all switches in resonable prices comes with only 4xQSFP+ or change my mind.

[interduo]

I saw be2e2fb
@thebracket do You know how big is performance penalty (and if is any) using only one interface instead of two?