ntop / libebpfflow

Container traffic visibility library based on eBPF

libebpfflow

Traffic visibility library based on eBPF

Introduction

libebpfflow is a traffic visibility library based on eBPF able to compute network flows. It can be used to:

• enable network visibility
• create a packet-less network probe
• inspect host and container communications for different container runtimes

Main features

• Ability to inspect TCP and UDP traffic
• Container visibility
• TCP latency computation
• Process and user visibility

Supported Languages

• Golang
• C/C++

Requirements

You need a modern eBPF-enabled Linux distribution.

On Ubuntu 16.04/18.04/20.04 Server LTS you can install the prerequisites (we assume that the compiler is already installed) as follows:

$ sudo apt-get install build-essential autoconf automake autogen libjson-c-dev pkg-config libzmq3-dev libcurl4-openssl-dev libbpfcc-dev

Build

Generate makefile

$ ./autogen.sh

Now build averything

$ make

Go testing tool

make go_ebpflowexport

Testing

The library comes with two different tools: ebpflowexport and go_ebpflowexport. In the Build section is reported how to build the tools. Although both tools were developed to show potential library usage and to provide guidance on how to use the library, ebpflowexport displays all the information provided by libebpfflow and provides some options for filtering flow events while go_ebpflowexport displays only basic information concerning events.

$ sudo ./ebpflowexport -h
ebpflowexport: Traffic visibility tool based on libebpfflow. By default all events will be shown 
Usage: ebpflow [ OPTIONS ] 
   -h, --help      display this message 
   -t, --tcp       TCP events 
   -u, --udp       UDP events 
   -i, --in        incoming events (i.e. TCP accept and UDP receive) 
   -o, --on        outgoing events (i.e. TCP connect and UDP send) 
   -r, --retr      retransmissions events 
   -c, --tcpclose  TCP close events 
   -d, --docker    gather additional information concerning containers (default: enabled)
   -v, --verbose   vebose formatting (default: every event is shown) 
Note: please run as root 

What follows is a demostration of the execution of ebpflowexport in a system where both minikube with containerd as runtime and docker containers are running at the same time.

$ sudo ./ebpflowexport -tio
Welcome to ebpflowexport v.1.0.190407
(C) 2018-19 ntop.org
Initializing eBPF [Legacy API]...
eBPF initializated successfully
1554803923.684786 [lo][Sent][IPv4/TCP][pid/tid: 1446/496 [/usr/bin/kubelet], uid/gid: 0/0][father pid/tid: 1/0 [/lib/systemd/systemd], uid/gid: 0/0][addr: 127.0.0.1:53790 <-> 127.0.0.1:10252][latency: 0.10 msec]
1554803923.685139 [lo][Rcvd][IPv4/TCP][pid/tid: 2554/2329 [/usr/local/bin/kube-controller-manager], uid/gid: 0/0][father pid/tid: 2295/0 [/usr/local/bin/containerd-shim], uid/gid: 0/0][addr: 127.0.0.1:53790 <-> 127.0.0.1:10252][containerID: 275d71585e03][runtime: containerd][kube_pod: kube-controller-manager-minikube][kube_ns: kube-system][latency: 0.00 msec]
1554803924.781354 [eth0][Sent][IPv4/TCP][pid/tid: 30197/30197 [/usr/bin/curl], uid/gid: 0/0][father pid/tid: 26219/0 [/bin/bash], uid/gid: 0/0][addr: 172.17.0.2:54348 <-> 216.58.205.46:80][containerID: cbd2540ec5be][runtime: docker][docker_name: sleepy_haibt][latency: 0.22 msec]
1554803929.257494 [enp0s3][Sent][IPv4/TCP][pid/tid: 30221/30221 [/usr/lib/apt/methods/http], uid/gid: 104/65534][father pid/tid: 30216/0 [/usr/bin/apt], uid/gid: 0/0][addr: 10.0.2.15:37140 <-> 91.189.88.162:80][latency: 0.17 msec]

A basic example of usage in c++ can be found in the directory /examples whereas for the Go language the example provided is the one in /go/ebpf_flow.go. More details on how to use the library you can be found in the ntopng code or by inspecting the code of the tool ebpflowexport application.

Export eBPF Information to ntopng

Supposing to start both ebpflowexport and ntopng on the same host do

• ntopng -i tcp://127.0.0.1:1234
• ebpflowexport -z tcp://127.0.0.1:1234

Start as a Docker container

To use ebpflowexport as a Docker container first you have to build the tool. Once the tool has been built, build the docker image from the project root:

$ docker build -t ebpflowexport .

The container can then be run

$ docker run -it --rm --privileged \
  -v /lib/modules:/lib/modules:ro \
  -v /usr/src:/usr/src:ro \
  -v /etc/localtime:/etc/localtime:ro \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /snap/bin/microk8s.ctr:/snap/bin/microk8s.ctr \
  ebpflowexport

Open Issues

While the library is already usable in production, we plan to add some additional features including:

• Implement periodic flow stats exports including bytes/packets/retransmissions

libebpfflow's Issues

error building go code

Error occurs when I run make go_ebpflowexport:

ebpf/bpf.go:222:44: filled_event.cgroup_id undefined (type *_Ctype_struct___0 has no field or method cgroup_id)
ebpf/bpf.go:225:74: filled_event.event undefined (type *_Ctype_struct___0 has no field or method event)
ebpf/bpf.go:231:74: filled_event.event undefined (type *_Ctype_struct___0 has no field or method event)
ebpf/bpf.go:238:27: filled_event.docker.dname undefined (type _Ctype_struct___2 has no field or method dname)
ebpf/bpf.go:240:50: filled_event.docker.dname undefined (type _Ctype_struct___2 has no field or method dname)

Obvious the golang code ebpf_flow.go using the structure eBPFevent but incomplete in c code.

typedef struct {
ktime_t ktime;
char ifname[IFNAMSIZ];
struct timeval event_time;
u_int8_t ip_version, sent_packet;
u16 etype;
union {
struct ipv4_addr_t v4;
struct ipv6_addr_t v6;
} addr;
u8 proto;
u16 sport, dport;
u32 latency_usec;
u16 retransmissions;
struct taskInfo proc, father;
char container_id[CONTAINER_ID_LEN];
struct {
char *name;
} docker;
struct {
char *name;
char *pod;
char *ns;
} kube;
} eBPFevent;

Compiling on Ubuntu 18 && ebpf_kprobe_attach_error

Hi,

I'm new to the library. Trying to compile on Ubuntu 18.04. Getting the error:

$ make
g++ -c -std=c++11 -g -Wall -I/usr/include/json-c -DHAVE_JSONC -I/usr/include/x86_64-linux-gnu -DHAVE_LIBCURL ebpf_flow.cpp -o ebpf_flow.o
In file included from /usr/include/bcc/BPF.h:25:0,
                 from ebpf_flow.cpp:32:
/usr/include/bcc/BPFTable.h:320:54: error: field ‘trace’ has incomplete type ‘bpf_stack_build_id [127]’
   struct bpf_stack_build_id trace[BPF_MAX_STACK_DEPTH];
                                                      ^
In file included from /usr/include/bcc/BPFTable.h:30:0,
                 from /usr/include/bcc/BPF.h:25,
                 from ebpf_flow.cpp:32:
/usr/include/bcc/bcc_syms.h:76:38: note: forward declaration of ‘struct bpf_stack_build_id’
                               struct bpf_stack_build_id *trace,
                                      ^~~~~~~~~~~~~~~~~~
Makefile:35: recipe for target 'libebpfflow.a' failed
make: *** [libebpfflow.a] Error 1

Running on Ubuntu 18:

$ uname -a
Linux osboxes 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic

BCC version:

$ apt list --installed | grep bcc
bcc/bionic,now 0.16.17-3.3 amd64 [installed]
bcc-tools/stable,stable,now 0.10.0-1 all [installed]
libbcc/stable,stable,now 0.10.0-1 all [installed,automatic]
libbcc-examples/stable,now 0.10.0-1 amd64 [installed]
libcc1-0/bionic-updates,bionic-security,now 8.3.0-6ubuntu1~18.04.1 amd64 [installed,automatic]
python-bcc/stable,stable,now 0.10.0-1 all [installed,automatic]

Installed BCC following instructions at https://github.com/iovisor/bcc/blob/master/INSTALL.md#ubuntu---binary

sudo apt-get install bpfcc-tools linux-headers-$(uname -r)

Found a similar closed issue at IOVisor: iovisor/bcc#2159. Solution there was to add include path to -I /usr/include/bcc/compat to g++.

I added include path to Makefile, and it then compiles. Though I'm getting following error when running ebpflowexport. (I added extra printfs in attachEBPFKernelProbe)

$ sudo ./ebpflowexport -tio
Welcome to ebpflowexport v.1.0.200130
(C) 2018-19 ntop.org
Initializing eBPF [Legacy API]...
[attachEBPFKernelProbe]: tcp_v4_connect/trace_connect_entry...
[attachEBPFKernelProbe]: tcp_v4_connect/trace_connect_v4_return...
ERROR: tcp_v4_connect/trace_connect_v4_return: -1
Unable to initialize libebpfflow: ebpf_kprobe_attach_error
eBPF terminated

Thanks,
Ruben

Add k8 integration

Currently k8 PODs are not reported by the library. Add k8 integration in addition to docker

Add ability to extract periodic network statistics

The library currently reports information about connections when they are created. It is requested to add a call that can be periodically called to poll the kernel and return connection stats such as bytes, packets, retransmission...

ctr ns ls issues

1.i deploy kubernetes in my cluster
2.deployment some services
3.use ctr ns ls just show moby, not k8s.io why？

Add flow termination handler

In addition to flow setup, we also need to generate an event when the flow ends. Hence it is requested to track flow end and report it

problem using libebpflow on ubuntu 19.10 (eoan ermine)

Hi,
I'm trying to use libebpfflow on ubuntu 19.10 ( eoan ermine ). I'm able to compile but
when I try to execute ebpflowexport (sudo ./ebpflowexport -tio) I get the same general error
"ebpf_initialization_failed" due to several compilation errors (12 errors total).
Any clues why this could happen ? Thanks in advance. Fausto.

Welcome to ebpflowexport v.1.0.191118
(C) 2018-19 ntop.org
Initializing eBPF [Legacy API]...
In file included from /virtual/main.c:22:
In file included from include/net/sock.h:46:
In file included from include/linux/netdevice.h:42:
In file included from include/net/netprio_cgroup.h:11:
In file included from include/linux/cgroup.h:28:
In file included from include/linux/cgroup-defs.h:22:
In file included from include/linux/bpf-cgroup.h:5:
include/linux/bpf.h:115:3: error: variable has incomplete type 'struct bpf_spin_lock'
(struct bpf_spin_lock){};
^~~~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:114:11: note: forward declaration of 'struct bpf_spin_lock'
*(struct bpf_spin_lock *)(dst + map->spin_lock_off) =
^
include/linux/bpf.h:125:22: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_spin_lock'
memcpy(dst + off + sizeof(struct bpf_spin_lock),
^ ~~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:125:36: note: forward declaration of 'struct bpf_spin_lock'
memcpy(dst + off + sizeof(struct bpf_spin_lock),
^
include/linux/bpf.h:126:22: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_spin_lock'
src + off + sizeof(struct bpf_spin_lock),
^ ~~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:125:36: note: forward declaration of 'struct bpf_spin_lock'
memcpy(dst + off + sizeof(struct bpf_spin_lock),
^
include/linux/bpf.h:127:34: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_spin_lock'
map->value_size - off - sizeof(struct bpf_spin_lock));
^ ~~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:125:36: note: forward declaration of 'struct bpf_spin_lock'
memcpy(dst + off + sizeof(struct bpf_spin_lock),
^
include/linux/bpf.h:454:39: error: use of undeclared identifier 'BPF_F_RDONLY_PROG'
u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
^
include/linux/bpf.h:454:59: error: use of undeclared identifier 'BPF_F_WRONLY_PROG'
u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
^
include/linux/bpf.h:459:21: error: use of undeclared identifier 'BPF_F_RDONLY_PROG'
if (access_flags & BPF_F_RDONLY_PROG)
^
include/linux/bpf.h:461:26: error: use of undeclared identifier 'BPF_F_WRONLY_PROG'
else if (access_flags & BPF_F_WRONLY_PROG)
^
include/linux/bpf.h:469:26: error: use of undeclared identifier 'BPF_F_RDONLY_PROG'
return (access_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) !=
^
include/linux/bpf.h:469:46: error: use of undeclared identifier 'BPF_F_WRONLY_PROG'
return (access_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) !=
^
include/linux/bpf.h:470:10: error: use of undeclared identifier 'BPF_F_RDONLY_PROG'
(BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
^
include/linux/bpf.h:470:30: error: use of undeclared identifier 'BPF_F_WRONLY_PROG'
(BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);

omissis
49 warnings and 12 errors generated.
Unable to initialize libebpfflow: ebpf_initialization_failed
eBPF terminated

libebpfflow not compiling Ubuntu 21.10 on Hyper V

Hello

Tryed to make the tool as explained on the README.

All the steps went smooth until i run make command :

jenubis@caperover:~/libebpfflow$ make
g++ -c -std=c++11 -g -Wall -I/usr/include/json-c -DHAVE_JSONC -I/usr/include/x86_64-linux-gnu -DHAVE_LIBCURL -fPIC ebpf_flow.cpp -o ebpf_flow.o
In file included from ebpf_flow.cpp:24:
container_info.h:32:11: error: ‘string’ in namespace ‘std’ does not name a type
   32 |      std::string name;
      |           ^~~~~~
container_info.h:29:1: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
   28 | #include <set>
  +++ |+#include <string>
   29 |
container_info.h:36:11: error: ‘string’ in namespace ‘std’ does not name a type
   36 |      std::string name, pod, ns;
      |           ^~~~~~
container_info.h:36:6: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
   36 |      std::string name, pod, ns;
      |      ^~~
container_info.h:51:27: error: ‘string’ is not a member of ‘std’
   51 |   std::unordered_map<std::string, struct cache_entry> gQueryCache;
      |                           ^~~~~~
container_info.h:51:27: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
container_info.h:51:27: error: ‘string’ is not a member of ‘std’
container_info.h:51:27: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
container_info.h:51:53: error: template argument 1 is invalid
   51 |   std::unordered_map<std::string, struct cache_entry> gQueryCache;
      |                                                     ^
container_info.h:51:53: error: template argument 3 is invalid
container_info.h:51:53: error: template argument 4 is invalid
container_info.h:51:53: error: template argument 5 is invalid
container_info.h:54:17: error: ‘string’ is not a member of ‘std’
   54 |   std::set<std::string> namespaces;
      |                 ^~~~~~
container_info.h:54:17: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
container_info.h:54:17: error: ‘string’ is not a member of ‘std’
container_info.h:54:17: note: ‘std::string’ is defined in header ‘<string>’; did you forget to ‘#include <string>’?
container_info.h:54:23: error: template argument 1 is invalid
   54 |   std::set<std::string> namespaces;
      |                       ^
container_info.h:54:23: error: template argument 2 is invalid
container_info.h:54:23: error: template argument 3 is invalid
ebpf_flow.cpp: In function ‘void ebpf_preprocess_event(eBPFevent*)’:
ebpf_flow.cpp:318:35: error: ‘struct container_info::<unnamed>’ has no member named ‘name’
  318 |         if(container_info->docker.name[0] != '\0') /* Docker info available */
      |                                   ^~~~
ebpf_flow.cpp:319:62: error: ‘struct container_info::<unnamed>’ has no member named ‘name’
  319 |           event->docker.name = strdup(container_info->docker.name.c_str());
      |                                                              ^~~~
ebpf_flow.cpp:321:33: error: ‘struct container_info::<unnamed>’ has no member named ‘name’
  321 |         if(container_info->kube.name[0] != '\0') /* Kubernetes info available */ {
      |                                 ^~~~
ebpf_flow.cpp:322:58: error: ‘struct container_info::<unnamed>’ has no member named ‘name’
  322 |           event->kube.name = strdup(container_info->kube.name.c_str());
      |                                                          ^~~~
ebpf_flow.cpp:323:58: error: ‘struct container_info::<unnamed>’ has no member named ‘pod’
  323 |           event->kube.pod  = strdup(container_info->kube.pod.c_str());
      |                                                          ^~~
ebpf_flow.cpp:324:58: error: ‘struct container_info::<unnamed>’ has no member named ‘ns’
  324 |           event->kube.ns   = strdup(container_info->kube.ns.c_str());
      |                                                          ^~
make: *** [Makefile:33: libebpfflow.a] Error 1

Is that normal ?

Regards

Thomas

how to fix "ebpf_initialization_failed"

hi, i have downloaded the newest libebpfflow , and install bcc 0.80.
but when i ran the ./ebpflowexport , i caught this error:

./ebpflowexport: /lib64/libldap_r-2.4.so.2: no version information available (required by /usr/local/lib/libcurl.so.4)
Welcome to ebpflowexport v.1.0.191103
(C) 2018-19 ntop.org
Initializing eBPF [Legacy API]...
In file included from :2:
/virtual/include/bcc/bpf.h:13:10: fatal error: 'linux/bpf_common.h' file not
found
#include <linux/bpf_common.h>
^~~~~~~~~~~~~~~~~~~~
1 error generated.
Unable to initialize libebpfflow: ebpf_initialization_failed
eBPF terminated

<command line>:4:9: note: previous definition is here #define __HAVE_BUILTIN_BSWAP32__ 1

  libebpfflow  git:(master) ✗ sudo ./ebpflowexport
Welcome to ebpflowexport v.1.0.230924
(C) 2018-22 ntop.org
Initializing eBPF [Legacy API]...
In file included from :2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:5:
In file included from include/linux/compiler_types.h:122:
include/linux/compiler-clang.h:41:9: warning: 'HAVE_BUILTIN_BSWAP32' macro redefined [-Wmacro-redefined]
#define HAVE_BUILTIN_BSWAP32
^
:4:9: note: previous definition is here
#define HAVE_BUILTIN_BSWAP32 1
^
In file included from :2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:5:
In file included from include/linux/compiler_types.h:122:
include/linux/compiler-clang.h:42:9: warning: 'HAVE_BUILTIN_BSWAP64' macro redefined [-Wmacro-redefined]
#define HAVE_BUILTIN_BSWAP64
^
:5:9: note: previous definition is here
#define HAVE_BUILTIN_BSWAP64 1
^
In file included from :2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:5:
In file included from include/linux/compiler_types.h:122:
include/linux/compiler-clang.h:43:9: warning: 'HAVE_BUILTIN_BSWAP16' macro redefined [-Wmacro-redefined]
#define HAVE_BUILTIN_BSWAP16
^
:3:9: note: previous definition is here
#define HAVE_BUILTIN_BSWAP16 1
^
/virtual/main.c:115:18: error: field has incomplete type 'struct timeval'
struct timeval event_time;
^
/virtual/main.c:115:10: note: forward declaration of 'struct timeval'
struct timeval event_time;
^
3 warnings and 1 error generated.
Unable to initialize libebpfflow: ebpf_initialization_failed
eBPF terminated

no data availabe in ntopng interface

the ntopng don't receive the data from the nprobe, no data available on ntopng interface web

this is my config:

ntopng:

-G=/var/run/ntopng.pid
--interface="tcp://127.0.0.1:5556
--https-port "3001"
--http-port "0"

nprob:

--zmq="tcp://*:5556"
--collector-port=2055
-n=none
-i=none

nprobe and ntponng are installed on the same server (ubuntu 16.04)

the nrpobe receive the data from my firewall

I have this error with ntopng

#ntopng -i tcp://127.0.0.1:5556

WARNING: Unable to open store /var/lib/ntopng/12/alerts/alerts_v7.db
[AlertsManager.cpp:942] ERROR: Unable to prepare statement for query SELECT count() FROM closed_alerts .
[AlertsManager.cpp:986] ERROR: Unable to prepare statement for query SELECT count() FROM flows_alerts .
[AlertsManager.cpp:942] ERROR: Unable to prepare statement for query SELECT count(*) FROM engaged_alerts WHERE alert_severity=2.
[Ntop.cpp:1996] Registered interface tcp://127.0.0.1:5556 [If Idx: 3] [id: 13]
[Ntop.cpp:1996] Registered interface tcp://127.0.0.1:5556 [If Idx: 4] [id: 14]

any idea please

thank you in advance