outscale / packetgraph

Packetgraph library is a collection of network bricks you can connect to form a network graph.

Packetgraph ?

Packetgraph is a library aiming to give the user a tool to build networks graph easily, It's built upon the fast DPDK library.

The goal of this library is to provide a really EASY interface to build you own DPDK based application using Network Function Virtualization Everyone is free to use this library to build up their own network application.

Once you have created and connected all bricks in you network graph, some bricks will be able to poll a burst of packets (max 64 packets) and let the burst propagate in you graph.

Connections between bricks don't store any packets and each burst will propagate in the graph without any copy.

Each graph run on one core but you can connect different graph using Queue bricks (which are thread safe). For example, a graph can be split on demand to be run on different core or even merged.

If you want a graphical representation of a graph, you can generate a dot output.

Available bricks (ipv4/ipv6):

• switch: a layer 2 switch
• rxtx: setup your own callbacks to get and sent packets
• tap: classic kernel virtual interface
• vhost: allow to connect a vhost NIC to a virtual machine (virtio based)
• firewall: allow traffic filtering passing through it (based on NPF)
• diode: only let packets pass in one direction
• accumulator: batch bursted packets to increase poll and burst speed
• hub: act as a hub device, passing packets to all connected bricks
• nic: allow passing packets to a NIC of the system (accelerated by DPDK)
• print: a basic print brick to show packets flowing through it
• antispoof: a basic mac checking, arp anti-spoofing and ipv6 neighbor discovery anti-spoofing
• vtep: VXLAN Virtual Terminal End Point switching packets on virtual LANs, can encapsulate packets over ipv4 or ipv6
• queue: temporally store packets between graph
• pmtud(ipv4 only): Path MTU Discovery is an implementation of RFC 1191
• user-dipole: setup your own callback in a dipole brick, to filter or implement your own protocol

A lot of other bricks can be created, check our wall ;)

How should I use Packetgraph ?

Code Documentation: doxygen link

Examples

To build and run examples, you may first check how to build Packetgraph below and adjust your configure command before make:

$ ./configure --with-examples
$ make

To run a specific example, check run scripts in tests directories:

$ ./examples/switch/run_vhost.sh
$ ./examples/switch/run.sh
$ ./examples/firewall/run.sh
$ ./examples/rxtx/run.sh
$ ./examples/dperf/run.sh
...

Building

You will need to build DPDK before building Packetgraph.

Install needed tools

You may adapt this depending on your Linux distribution:

Ubuntu

$ sudo apt-get install libpcap-dev libglib2.0-dev libjemalloc-dev libnuma-dev openssl

CentOS

$ sudo yum install -y glibc-devel glib2-devel libpcap-devel git wget numactl numactl-devel openssl-devel clang
$ wget http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-devel-3.6.0-8.el7.centos.x86_64.rpm
$ wget http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-3.6.0-8.el7.centos.x86_64.rpm
$ sudo rpm -i jemalloc-devel-3.6.0-8.el7.centos.x86_64.rpm jemalloc-3.6.0-8.el7.centos.x86_64.rpm

Build DPDK

$ git clone http://dpdk.org/git/dpdk
$ cd dpdk
$ git checkout -b v19.02 v19.02
$ make config T=x86_64-native-linuxapp-gcc

Note: use T=x86_64-native-linuxapp-clang to build with clang

Edit build/.config and be sure to set the following parameters to 'y':

• CONFIG_RTE_LIBRTE_PMD_PCAP

If you don't want to use some special PMD in DPDK requiring kernel headers, you will have to set the following parameters to 'n':

• CONFIG_RTE_EAL_IGB_UIO
• CONFIG_RTE_KNI_KMOD

Once your .config file is ready, you can now build dpdk as follows:

$ make EXTRA_CFLAGS='-fPIC'

Finally, set RTE_SDK environment variable:

$ export RTE_SDK=$(pwd)

Build packetgraph

$ git clone https://github.com/outscale/packetgraph.git
$ cd packetgraph
$ git submodule update --init
$ ./configure
$ make
$ make install

Note: to build with clang, you can use ./configure CC=clang

Note 2: You need a compiler that support C11 (gcc 4.9 or superior, or clang 3.4 or superior).

Configure Huge Pages

Packetgraph uses some huge pages (adjust to your needs):

• Edit your /etc/sysctl.conf and add some huge pages:

vm.nr_hugepages=2000

• Reload your sysctl configuration:

$ sudo sysctl -p /etc/sysctl.conf

• Check that your huge pages are available:

$ cat /proc/meminfo | grep Huge

• Mount your huge pages:

$ sudo mkdir -p /mnt/huge
$ sudo mount -t hugetlbfs nodev /mnt/huge

• (optional) Add this mount in your /etc/fstab:

hugetlbfs       /mnt/huge  hugetlbfs       rw,mode=0777        0 0

Compille Time Optimisation:

-DPG_BRICK_NO_ATOMIC_COUNT: do not use atomic variable to count packets, if you do so, you must call pg_brick_pkts_count_get in the same thread you use to poll packets -DPG_VHOST_FASTER_YET_BROKEN_POLL: change the way vhost lock the queue so it spend less time locking/unlocking the queue, but can easily deadlock if badly use.

Compille Time Option

-DTAP_IGNORE_ERROR: when packetgrapg can't burst or poll a tap, it return 0 instead of returning an error -DPG_QEMU_MEM_SIZE=SIZE: allow to specify qemy ram size in test, parameter must be a string in qemu format, examples: -DPG_QEMU_MEM_SIZE="1G", -DPG_QEMU_MEM_SIZE="512M"

Licence

Packetgraph project is published under GNU GPLv3. For more information, check LICENSE file.

Contribute

New to packetgraph ? Want to contribute and/or create a new brick ? Some developer guidelines are available.

Question ? Contact us !

Packetgraph is an open-source project, feel free to chat with us on IRC

server: irc.freenode.org

chan: #betterfly

packetgraph's Issues

wall: queue brick to ease multithreading

This new brick permits to buffer packets in a queue until they are polled.
This brick should have the "poll" flag enabled so it can be shared in several graphic structure (see #18 )

vtep: manage maximal MTU size

We sould be able to configure the max MTU managed by vtep once packets are encapsulated.
vtep should drop packets if it cannot encapsulate them and respect the MTU

nic: test link connect/disconnect

We just have to check that link connection / disconnection should not break anything

vhost: unregister at destroy

We should call rte_vhost_driver_unregister during vhost destroy once this function is implemented

core: brick max edge could be zero

Allow to create brick with “*_max” parameter equal to 0.
For the moment, this produce a crash

nic: select a system nic by it's name is broken

Rename brick-*

Just rename brick-XXX to XXX, easier to type ;)

print: manage VXLAN payload in UDP print

Show VXLAN headers (vxlan id) in print brick and call printer_eth() again to show vxlan payload

vtep: remove parse_args from tests

vtep: segment array for vtep

A vtep have an array of sides where each edge correspond to a VNI.

A VNI is nothing more than a short int.

So we could have a faster way address our packets to the good VNI if we had a segmented array
use to made 1-1 mapping between VNI and brick's edge instead of looping on each edge and see if the packets we are transmitting are corresponding to the edge we are looping on.

Clean urcu usage in project

We have a lot of urcu references which should not be here:
https://github.com/outscale/packetgraph/search?utf8=%E2%9C%93&q=urcu

We should clean this

Move cmake directories

We should move cmake include directory in the root

core: brick link takes a variable list of bricks to link

The idea is to link more than 2 bricks together.

core: create a graph stucture to contain a whole graph

The idea to put all your linked bricks inside a graph.
Once populated on this structure, the user could call a poll() function to poll all pollables bricks of the graph.
To know if a brick is pollable or not, we could add a flag in each brick.
Each time the poll function of the graph is called, an integer "counter" is incremented.

Then, when looking for bricks during the poll:

• 0: the brick is not pollable
• 1: the brick will be pulled each time
• 2: the brick will be pulled when counter % 2 = 0
• 3: the brick will be pulled when counter % 3 = 0
• ...

build: make the project build with clang

Nice to have :)

Deduplicate pg_packets_prepare_hash_keys in bricks

vtep call pg_packets_prepare_hash_keys.
switch call pg_packets_prepare_hash_keys.
antispoof can be faster if the packets was prepare by pg_packets_prepare_hash_keys.

We should call pg_packets_prepare_hash_keys after the nic brick, of create a brick to call pg_packets_prepare_hash_keys and made an option to stop doing that in switch.
After that we can optimize the antispoof, and all other brick which check the ether_addr.

vhost: replace the brick by pmd

It's a reminder: a pmd wrapping vhost is beeing reviewed on dpdk.

When this pmd will be ready, there would be no more reason to maintain a vhost brick instead of using the nic brick.
The main work to do once this feature ready would be to dynamically create nic devices in nic brick

move documentation in headers

A lot of header documentation lives in source code.
We should move it to headers.

print: use layers to parse protocols

Instead of looking at all possible protocols of layer n+1 in layer n, we should use a common function for each layers.
If n+1 fails to parse, we can try to look at layer n and so on (useful for lower encapsulated layers)

core: remove brick_handle_dup

Prefix public functions

We sould prefix public functions: pg_* ?

nic: copy stats field by field

We use a copy of nic stats, to avoid braking future version during copy we should copy field by field them.
Another option is to expose the original stat structure

core: thread monitoring documentation

Add some doc about how to monitor threads in packetgraph.

vhost: get stats about packet

This will be usefull to have some stats about traffic passing through the vhost brick.
An other solution would be to have a separate statistic brick.

vhost: optimization using private data

We should use vhost's private data to store our brick.

When a qemu crash, the vhost brick crash too.

So we need to fix that and create a test sending a segv to a running qemu.

Can only reload the firewall in the same thread who created it.

If we create the firewall brick in a thread and reload it in another thread, this will block firewall_reload.

core: some bricks should not be allowed nor able to forward to more than one brick

Today, we can classify our brick in 2 categories: the social brick which are dealing with lot of neighbors(hub, switch vtep) and the associable one(diode, firewall, nic, vhost...), which are not make to deal with more that 2 brick(one on each side) and actually if we try to link an associable brick with a lot of neighbor, I'm not really sure about what will happen, and this can cause bugs.

We need to forbid associable bricks to deal with more than 1 neighbor on each side, and so we will be able to optimize the forwarding of the associable bricks.

Allow to build without building tests

Some brick have tests which have more dependency than the brick himself(hello vtep and vhost).

vtep: use rte_eth_vxlan_decap and rte_eth_vxlan_encap

rte_eth_vxlan_decap and rte_eth_vxlan_encap are new features of rte_ether.
We should use this in vtep brick once implemented in dpdk

Check all headers

Just a reminder to check all IFNDEF headers and check that they respect naming conventions

Returns logic

We have to check return logic: some functions returns 1 if ok, 0 otherwise and some have the opposite behavior.

Homogenize network structures

Some file use dpdk function to manipulate Ethernet some other use system include, we need to homogenize that.

core: count links of a brick

We already have functions to count this but we should make them public

free hugepage

When an application launch and stops VMs who connect to packetgraph using vhost we can see that Hugepages are never freed. I wrote a test to reproduce this behavior in a separate branch.

vhost: improve debug when test crash.

Because, we had a lot of problem with this test, and have no information about where the bug come from.

core: rename switch_error

switch_error is an old structure who should be renamed pg_error

core: clean packetsgen

Remove the "3 packets" generator and just provide a function to generate them.

collect: use rte_pktmbuf_clone instead of memcpy in burst

Clean urcu usage in project

We have a lot of urcu references which should not be here:
https://github.com/outscale/packetgraph/search?utf8=%E2%9C%93&q=urcu

We should clean this

wall: linux container brick

The idea is to create a new brick permitting to communicate with a linux container.
This feature is currently under specification (see this thread).

core: packetgraph_start() return

packetgraph_start() return is quite obscure, we should just return 0 or 1 if the start was successfull or not.

Somes brick don't check if config is NULL and somes orther may overchek it

the clasical way to create a brick look like this:

mybrick_new(args)
{
  config = mybrick_config_new(args)
  if (!config)
     return NULL;
  ret = brick_new("mybrick", config, errp);
  brick_config_free(config);
  return ret;
}

Exept that some brick didn't check if config is null,
and some other check if config is NULL inside "mybrick_init"
which is useless because if config is NULL, you'll get a SEGV before reaching
mybrick_init.
We need to homogenize this code.

vhost: get socket path function

Would be usefull to have a public function in vhost brick to return socket path

Cache before calling map.

Lookup in a map is slow.
Add an element to a map, include a lookup inside the map, it's slow.

So we should add a cache system, using a little circular static array, which will allow us to
to skip calling the map functions.

Complete documentation about dpdk building

We should complete the documentation on dpdk building.

For example, we should add that we must install kernel dev tools and kernel headers

core: test that forwarding to ... nowhere don't break.

When forwarding to a dead endpoint, this should not break. We just have to ensure that is safe to do this.

vtep: deduplicate code from multicast_subscribe/multicast_unsubscribe

The code use to build ip/mac/udp and igmp header are mostly the same.
It should be a good idea to centralize them.
Maybe create helper functions in core/packetgraph/utils/

firewall spawn limitation

In tests (integration), spawning firewall instances are limited and stops around 62.
We need to investigate this.

core: able to unlink a specific side only

It would be nice to be able to only unlink a specific side of a brick.