rjarry / brouter

grout # a graph router based on DPDK

grout # a graph router based on DPDK

grout stands for Graph Router. In English, "grout" refers to thin mortar that hardens to fill gaps between tiles.

grout is a DPDK based network processing application. It uses the rte_graph library for data path processing.

Its main purpose is to simulate a network function or a physical router for testing/replicating real (usually closed source) VNF/CNF behavior with an opensource tool.

It comes with a client library to configure it over a standard UNIX socket and a CLI that uses that library. The CLI can be used as an interactive shell, but also in scripts one command at a time, or by batches.

License

BSD-3-Clause

Features

Supported

• IPv4 basic forwarding
• ARP resolution/reply (packets waiting for resolution are buffered)
• ICMP echo reply
• Multiple VRF domains
• VLAN sub interfaces
• IP in IP tunnels

In Progress

• IPv6 forwarding
• NDP resolution/reply

Planned Short Term

• FRR zebra forwarding plane plugin
• loopback interfaces
• packet tracing

Planned Long Term

• hardware offload using generic DPDK APIs
• IPsec datapath
• strongSwan plugin for IKEv2 integration
• port mirroring for tcpdump native support
• ...

Dependencies

Name	Type	License	Code
DPDK	Build & Runtime	BSD-3-Clause	https://git.dpdk.org/dpdk/
libnuma	Build & Runtime	LGPL-2.1	https://github.com/numactl/numactl
libevent	Build & Runtime	BSD-3-Clause	https://github.com/libevent/libevent
libstb	Build & Runtime	Public Domain	https://github.com/nothings/stb
libecoli	Build & Runtime	BSD-3-Clause	https://git.sr.ht/~rjarry/libecoli
libsmartcols	Build & Runtime	LGPL-2.1	https://github.com/util-linux/util-linux/tree/master/libsmartcols
cmocka	Build	Apache-2.0	https://github.com/clibs/cmocka
meson	Build	Apache-2.0	https://github.com/mesonbuild/meson
ninja	Build	Apache-2.0	https://github.com/ninja-build/ninja
libasan	Dev	MIT+BSD	https://github.com/gcc-mirror/gcc/tree/master/libsanitizer
clang-format	Dev	MIT+BSD	https://clang.llvm.org/docs/ClangFormat.html

Quick start

Install system dependencies

dnf install gcc git libcmocka-devel libedit-devel libevent-devel make meson \
        ninja-build numactl-devel pkgconf python3-pyelftools scdoc \
        libsmartcols-devel

or

apt install build-essential gcovr libcmocka-dev libedit-dev libevent-dev \
        libnuma-dev meson ninja-build pkg-config python3-pyelftools scdoc \
        libsmartcols-dev

Build

git clone https://github.com/rjarry/grout
cd grout
make

Start the router daemon

[root@dio grout]$ taskset --cpu-list 6-19,26-39 ./build/grout -v
GR: dpdk_init: DPDK 24.03.0
GR: dpdk_init: EAL arguments: -l 0 -a 0000:00:00.0 --log-level=*:notice --log-level=gr:info
EAL: Detected CPU lcores: 40
EAL: Detected NUMA nodes: 1
...
GROUT: listen_api_socket: listening on API socket /run/grout.sock

Start the CLI

By default, the CLI will start an interactive shell with command completion:

[root@dio grout]$ ./build/grcli
Welcome to the grout CLI.
Use ? for help and <tab> for command completion.
grout# ?
quit                 Exit the CLI.
add                  Create objects in the configuration.
del                  Delete objects from the configuration.
show                 Display information about the configuration.
clear                Clear counters or temporary entries.
set                  Modify existing objects in the configuration.

Multiple commands can be piped into standard input:

[root@dio grout]$ ./build/grcli -ex < commands.list
+ add interface port p0 devargs 0000:18:00.0 rxqs 1 qsize 2048
Created interface 0
+ add interface port p1 devargs 0000:18:00.1 rxqs 1 qsize 2048
Created interface 1
+ set port qmap p0 rxq 0 cpu 7
+ set port qmap p1 rxq 0 cpu 27
+ show interface all
NAME  ID  FLAGS       TYPE  INFO
p0    0   up running  port  devargs=0000:18:00.0 mac=b8:3f:d2:fa:53:86
p1    1   up running  port  devargs=0000:18:00.1 mac=b8:3f:d2:fa:53:87
+ show port qmap
IFACE  RXQ_ID  CPU_ID  ENABLED
p0     0       7       1
p1     0       27      1
+ add ip address 172.16.0.2/32 iface p0
+ add ip address 172.16.1.2/32 iface p1
+ add ip nexthop 172.16.0.1 mac b8:3f:d2:fa:53:7a iface p0
+ add ip nexthop 172.16.1.1 mac b8:3f:d2:fa:53:7b iface p1
+ add ip route 172.16.0.0/24 via 172.16.0.1
+ add ip route 172.16.1.0/24 via 172.16.1.1
+ add ip route 0.0.0.0/0 via 172.16.1.183
+ show ip address
IFACE  ADDRESS
p0     172.16.0.2/32
p1     172.16.1.2/32
+ show ip route
DESTINATION    NEXT_HOP
172.16.0.1/32  172.16.0.1
172.16.0.2/32  172.16.0.2
172.16.0.0/24  172.16.0.1
172.16.1.1/32  172.16.1.1
172.16.1.2/32  172.16.1.2
172.16.1.0/24  172.16.1.1
0.0.0.0/0      172.16.1.183
+ show ip nexthop
IP            MAC                IFACE  AGE  STATE
172.16.1.183  ??:??:??:??:??:??  ?      ?    gateway
172.16.1.2    b8:3f:d2:fa:53:87  p1     0    reachable static local link
172.16.0.2    b8:3f:d2:fa:53:86  p0     0    reachable static local link
172.16.0.1    b8:3f:d2:fa:53:7a  p0     0    reachable static gateway
172.16.1.1    b8:3f:d2:fa:53:7b  p1     0    reachable static gateway

The CLI can be used as a one-shot command (with bash completion built-in):

[root@dio grout]$ complete -o default -C './build/grcli -c' ./build/grcli
[root@dio grout]$ ./build/grcli <TAB><TAB>
add                 (Create objects in the configuration.)
clear               (Clear counters or temporary entries.)
del                 (Delete objects from the configuration.)
-e                  (Abort on first error.)
--err-exit          (Abort on first error.)
--help              (Show usage help and exit.)
-h                  (Show usage help and exit.)
quit                (Exit the CLI.)
set                 (Modify existing objects in the configuration.)
show                (Display information about the configuration.)
--socket            (Path to the control plane API socket.)
-s                  (Path to the control plane API socket.)
--trace-commands    (Print executed commands.)
-x                  (Print executed commands.)
[root@dio grout]$ ./build/grcli show <TAB><TAB>
graph        (Show packet processing graph info.)
interface    (Display interface details.)
ip           (Show IPv4 stack details.)
port         (Display DPDK port information.)
stats        (Print statistics.)
[root@dio grout]$ ./build/grcli show stats <TAB><TAB>
hardware    (Print hardware stats.)
software    (Print software stats.)
[root@dio grout]$ ./build/grcli show stats software
NODE         CALLS   PACKETS  PKTS/CALL  CYCLES/CALL  CYCLES/PKT
port_rx     757792  22623757       29.9       1776.4        59.5
ip_input    333675  22623757       67.8       3091.0        45.6
port_tx     333675  22623757       67.8       1984.2        29.3
eth_input   757792  22623757       29.9        659.7        22.1
eth_output  333675  22623757       67.8       1323.4        19.5
ip_output   333675  22623757       67.8        926.3        13.7
ip_forward  333675  22623757       67.8        691.8        10.2

Packet graph

[root@dio grout]$ ./build/grcli show graph dot | dot -Tsvg > docs/graph.svg

brouter's Issues

add multiple vrf support

mbuf data pointer shifting across the graph

Can it have any performance impact?

allow configuring interfaces with local IP addresses

implement ARP request/reply in the data path

add ipv6 routing support

integrate rcu for rte_fib when available upstream

More details: http://inbox.dpdk.org/dev/[email protected]/

Implement IP in IP interfaces

implement icmp rate limiting

patch upstream dpdk nodes to support dynamic configuration

add logical interfaces abstraction

implement nexthop/arp refreshing

We need a way to keep the nexthop cache up to date by sending periodic arp requests.

write architecture document

implement port mirroring

Allow mirroring traffic from a port to a linux tap interface so that we can run regular tcpdump on it.

Example:

[root@dio brouter]$ br-cli mirror add tap-port0 port 0
[root@dio brouter]$ tcpdump -nni tap-port0 -c 5
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tap-port0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:36:31.483702 IP 192.168.1.33.5353 > 224.0.0.251.5353: 0*- [0q] 1/0/0 (Cache flush) TXT "deviceid=DC:A6:32:CC:CD:37" "model=Xbmc,1" "srcvers=101.28" "features=0x20F7" (131)
14:36:31.758020 IP6 2a01:cb00:f8b:9700:ead2:ffff:fee4:99e0.53 > 2a01:cb00:f8b:9700:ffd8:872f:c4a:d9d4.35798: 27883 2/0/1 CNAME edge-web.dual-gslb.spotify.com., AAAA 2600:1901:1:c36:: (124)
14:36:31.759541 IP6 2a01:cb00:f8b:9700:ead2:ffff:fee4:99e0.53 > 2a01:cb00:f8b:9700:ffd8:872f:c4a:d9d4.49708: 30401 2/0/1 CNAME edge-web.dual-gslb.spotify.com., A 35.186.224.25 (112)
14:36:32.507441 ARP, Request who-has 192.168.1.1 (ff:ff:ff:ff:ff:ff) tell 192.168.1.16, length 46
14:36:33.324903 IP6 2001:41d0:a:69e1::1.1107 > 2a01:cb00:f8b:9700:ffd8:872f:c4a:d9d4.40526: Flags [P.], seq 4102294619:4102294705, ack 3835940537, win 501, options [nop,nop,TS val 1300040159 ecr 4009689886], length 86
5 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@dio brouter]$ br-cli mirror del tap-port0

Implement vlan sub interface support

write man pages for br and br-cli

implement a basic frr forwarding plane plugin

https://docs.frrouting.org/en/latest/zebra.html#zebra-fib-push-interface

Or can we implement a loadable plugin that will directly interact with libbr.so and not have any netlink/protobuf overhead?

support interfaces with fixed number of tx queues (vhost/virtio)

Some interfaces have limited (fixed sometimes) number of tx queues. We cannot force one tx queue per datapath worker thread.

Potential solutions:

• Use graph pipeline mode
• Use locks for these specific tx queues

All these solutions will affect the overall performance of the other ports.