milgaluppang / awesome-ebpf

Awesome eBPF

A curated list of awesome projects related to eBPF.

Note: eBPF is an exciting piece of technology, and it's ecosystem is constantly evolving. We'd love help from you to keep this awesome list up to date, and improve its signal-to-noise ratio in anyway we can. Please feel free to leave any feedback.

Contents

• What is BPF?
• Resources
• Projects based on, or related to eBPF
• Tutorials
• Examples
• The Code
• Tools and Utilities
• Development and Community
• Other lists of resources on eBPF
• Acknowledgement
• Contributing To This List
• License

What is BPF?

TODO: Update with concise overview of BPF (cBPF and eBPF), and what it's used for already.

Resources

Generic Documentation and Presentations

If you are new to eBPF, you may want to try the links described as “introduction” or ”documentation” in this section (although you might not want to start with “kernel documentation”, which is dense).

• linux/Documentation/networking/filter.txt. Kernel documentation: eBPF specification (somewhat outdated; information should still be valid, but not exhaustive).
• linux/Documentation/bpf/bpf_design_QA.rst. Kernel documentation: Frequently Asked Questions on eBPF design.
• IO Visor's Unofficial eBPF spec Summary of eBPF syntax and operation codes.
• Manual pages
  • bpf(2) man page about the bpf() system call, used to manage BPF programs and maps from userspace.
  • tc-bpf(8) man page about using BPF with tc, including example commands and samples of code.
• Jesper Dangaard Brouer's documentation: work in progress, contributions welcome.
• Cilum's BPF and XDP Reference Guide Generic documentation about most features of eBPF.
• Emails from David Miller to the xdp-newbies mailing list:
  • bpf.h and you…
  • Contextually speaking…
  • BPF Verifier Overview
• A blog post series about eBPF from Ferris Ellis.
• List of BPF features per kernel version, in bcc repository.
• A BPF reference guide about BPF C and bcc Python helpers, from bcc repository.
• Making the Kernel’s Networking Data Path Programmable with BPF and XDP (Daniel Borkmann, OSSNA17, Los Angeles, September 2017) A set of slides covering all the basics about eBPF and XDP (mostly for network processing).
• The BSD Packet Filter (Suchakra Sharma, June 2017) An introduction mostly covering the tracing aspects.
• BPF: tracing and more (Brendan Gregg, January 2017) An introduction mostly covering the tracing aspects.
• Linux BPF Superpowers (Brendan Gregg, March 2016) An introduction mostly covering the tracing aspects, first part with flame graphs.
• IO Visor (Brenden Blanco, SCaLE 14x, January 2016) Also introduces IO Visor project.
• BPF — in-kernel virtual machine (Alexei Starovoitov, February 2015) Presentation by the author of eBPF.
• Extending extended BPF (Jonathan Corbet, July 2014).

BPF Internals

• Daniel Borkmann has made several presentations and papers covering the internals of eBPF, in particular about its use with tc.
  • eBPF and XDP walkthrough and recent updates (fosdem17, Brussels, Belgium, February 2017).
  • Advanced programmability and recent updates with tc's cls_bpf (netdev 1.2, Tokyo, October 2016) Details on eBPF, its use for tunneling and encapsulation, direct packet access, and more.
  • cls_bpf/eBPF updates since netdev 1.1 (netdev 1.2, Tokyo, October 2016, part of this tc workshop).
  • On getting tc classifier fully programmable with cls_bpf (netdev 1.1, Sevilla, February 2016) Introduction to eBPF, including several features (map management, tail calls, verifier). The full paper is also available here.
  • Linux tc and eBPF (fosdem16, Brussels, Belgium, January 2016).
• IO Visor blog.
• Linux Networking Explained (Thomas Graf, LinuxCon, Toronto, August 2016) Linux networking internals, with a part about eBPF.

Kernel Tracing

• Meet-cute between eBPF and Kernel Tracing (Viller Hsiao, July 2016) Kprobes, uprobes, ftrace
• Linux Kernel Tracing (Viller Hsiao, July 2016) Systemtap, Kernelshark, trace-cmd, LTTng, perf-tool, ftrace, hist-trigger, perf, function tracer, tracepoint, kprobe/uprobe…
• Brendan Gregg's blog, and in particular Linux BPF Superpowers article.

XDP

• Work-in-progress documentation for XDP started by Jesper Dangaard Brouer, meant to be a collaborative work; contributions welcome.
• The BPF and XDP Reference Guide from Cilium project.
• XDP overview on the IO Visor website.
• eXpress Data Path (XDP) (Tom Herbert, Alexei Starovoitov, March 2016) The first presentation about XDP.
• BoF - What Can BPF Do For You? (Brenden Blanco, LinuxCon, Toronto, August 2016).
• eXpress Data Path (Brenden Blanco, Linux Meetup at Santa Clara, July 2016) Contains some benchmark results obtained with the mlx4 driver.
• Jesper Dangaard Brouer has several sets of slides describing the internals of XDP:
  • XDP − eXpress Data Path, Intro and future use-cases (September 2016) “Linux Kernel’s fight against DPDK”. Future plans (as of this writing) for XDP and comparison with DPDK.
  • Network Performance Workshop (netdev 1.2, Tokyo, October 2016) Additional hints about XDP internals and expected evolution.
  • XDP – eXpress Data Path, Used for DDoS protection (OpenSourceDays, March 2017) Details and use cases about XDP, with benchmark results, and code snippets for benchmarking as well as for basic DDoS protection with eBPF/XDP (based on an IP blacklisting scheme).
  • Memory vs. Networking, Provoking and fixing memory bottlenecks (LSF Memory Management Summit, March 2017) Advanced details about current memory issues faced by XDP developers.
  • XDP for the Rest of Us (netdev 2.1, Montreal, April 2017), with Andy Gospodarek. How to get started with eBPF and XDP for normal humans. Also summarized by Julia Evans on her blog.
• XDP workshop — Introduction, experience, and future development (Tom Herbert, netdev 1.2, Tokyo, October 2016) (Video).
• High Speed Packet Filtering on Linux (Gilberto Bertin, DEF CON 25, Las Vegas, July 2017) About packet filtering on Linux, DDoS protection, packet processing in the kernel, kernel bypass, XDP and eBPF.

cBPF

• The BSD Packet Filter: A New Architecture for User-level Packet Capture (Steven McCanne and Van Jacobson, 1992) The original paper about (classic) BPF.
• The FreeBSD manual page about BPF.
• Linux’ packet mmap(2), BPF, and Netsniff-NG (Daniel Borkmann, 2013).
• tc and cls bpf: lightweight packet classifying with BPF (Daniel Borkmann, 2013).
• Introducing Cloudflare's BPF Tools (Marek Majkowski, Cloudflare, 2014) Usage of BPF bytecode with the xt_bpf module for iptables.
• Libpcap filters syntax.

Hardware Offload

• eBPF/XDP hardware offload to SmartNICs (Jakub Kicinski and Nic Viljoen, netdev 1.2, Tokyo, October 2016) Hardware offload for eBPF with TC or XDP (Linux kernel 4.9+), introduced by Netronome.

Projects based on, or related to eBPF

• P4 has some interactions with eBPF:
  • P4 on the Edge (John Fastabend, May 2016) P4 with eBPF to create high-performance programmable switches.
  • OvS Orbit episode (#11), called P4 on the Edge, (August 2016), related to the former item. Audio interview of John Fastabend by Ben Pfaff, one of the core maintainers of Open vSwitch.
  • P4, EBPF and Linux TC Offload (Dinan Gunawardena and Jakub Kicinski, August 2016) P4 with some elements related to eBPF hardware offload on Netronome's NFP (Network Flow Processor) architecture.
  • Old documentation for P4 usage with eBPF, from bcc repository; deprecated by the P4_16 backend linked below.
  • P4_16 backend for eBPF.
• Cilium project (GitHub repository) is a technology relying on BPF and XDP to provide “fast in-kernel networking and security policy enforcement for containers based on eBPF programs generated on the fly”. Many presentations available (with overlap):
  • Cilium: Networking & Security for Containers with BPF & XDP, also featuring a load balancer use case (Thomas Graf, Linux Plumbers conference, Santa Fe, November 2016)
  • Cilium: Networking & Security for Containers with BPF & XDP (Thomas Graf, Docker Distributed Systems Summit, October 2016 — video)
  • Cilium: Fast IPv6 container Networking with BPF and XDP (Thomas Graf, LinuxCon, Toronto, August 2016)
  • Cilium: BPF & XDP for containers (Thomas Graf, fosdem17, Brussels, Belgium, February 2017)
  • OvS Orbit episode (#4) (May 2016) Interview of Thomas Graf by Ben Pfaff.
  • A generic introduction to Cilium (Daniel Borkmann, as a guest author on Google Open Source blog, November 2016).
  • A podcast by Ivan Pepelnjak by Ivan Pepelnjak interviewing Thomas Graf (October 2016) on eBPF, P4, XDP and Cilium.
• Open vSwitch (OvS), and its related project Open Virtual Network (OVN, an open source network virtualization solution) are considering to use eBPF at various level:
  • Offloading OVS Flow Processing using eBPF (William (Cheng-Chun) Tu, OvS conference, San Jose, November 2016)
  • Coupling the Flexibility of OVN with the Efficiency of IOVisor (Fulvio Risso, Matteo Bertrone and Mauricio Vasquez Bernal, OvS conference, San Jose, November 2016)
• XDP in practice: integrating XDP in our DDoS mitigation pipeline (Gilberto Bertin, netdev 2.1, Montreal, April 2017) Protection against DDoS with XDP at Cloudflare.
• Droplet: DDoS countermeasures powered by BPF + XDP (Huapeng Zhou, Doug Porter, Ryan Tierney, Nikita Shirokov, netdev 2.1, Montreal, April 2017) Protection against DDoS with XDP at Facebook.
• CETH for XDP (Yan Chan and Yunsong Lu, Linux Meetup, Santa Clara, July 2016) Common Ethernet Driver Framework for faster network I/O, a technology initiated by Mellanox.
• The VALE switch has a BPF extension module.
• Suricata, an open source intrusion detection system, relies on eBPF components for its “capture bypass” features:
  • The adventures of a Suricate in eBPF land (Éric Leblond, netdev 1.2, Tokyo, October 2016)
  • eBPF and XDP seen from the eyes of a meerkat (Éric Leblond, Kernel Recipes, Paris, September 2017)
• InKeV: In-Kernel Distributed Network Virtualization for DCN (Z. Ahmed, M. H. Alizai and A. A. Syed, SIGCOMM, August 2016)
• gobpf - utilizing eBPF from Go (Michael Schubert, fosdem17, Brussels, Belgium, February 2017) A “library to create, load and use eBPF programs from Go”
• ply A small but flexible open source dynamic tracer for Linux, with features similar to the bcc tools, but with a simpler language inspired by awk and dtrace.
• bpfd Framework for running BPF programs with rules on Linux as a daemon. Container aware.

Tutorials

• bcc Reference Guide: many incremental steps to start using bcc and eBPF, mostly centered on tracing and monitoring.
• bcc Python Developer Tutorial: also comes with bcc, but targets the Python bits across seventeen “lessons”.
• Linux Tracing Workshops Materials from Sasha Goldshtein: involves the use of several BPF tools for tracing.
• Tracing a packet journey using Linux tracepoints, perf and eBPF from Jean-Tiare Le Bigot: troobleshooting ping requests and replies with perf and bcc programs.
• Open NFP platform operated by Netronome: some tutorials for network-related eBPF use cases, including an eBPF Offload Starting Guide.
• XDP for the Rest of Us from Jesper Dangaard Brouer and Andy Gospodarek at Netdev 2.1: first edition of a workshop to get started with XDP.
• XDP for the Rest of Us from the same authors, at Netdev 2.2: second edition, with new contents.

Examples

• linux/samples/bpf/ in the kernel tree: some sample eBPF programs.
• linux/tools/testing/selftests/bpf in the kernel tree: Linux BPF selftests, with many eBPF programs.
• prototype-kernel/kernel/samples/bpf from Jesper Dangaard Brouer's prototype-kernel repository contains some additional examples that can be compiled outside of kernel infrastructure.
• iproute2/examples/bpf/ from iproute2 package: some networking programs to attach to the TC interface.
• bcc/examples: coming along with the bcc tools, mostly about tracing.
• bcc/tools themselves can be seen as example use cases for BPF programs, mostly for tracing and monitoring. bcc tools have been packaged for some Linux distributions.
• MPLSinIP sample a heavily commented sample demonstrating how to encapsulate & decapsulate MPLS within IP. The code is commented for those new to BPF development.

The Code

• linux/include/linux/bpf.h, linux/include/uapi/bpf.h: definitions related to eBPF, to be used respectively in the kernel and to interface with userspace programs.
• linux/include/linux/filter.h, linux/include/uapi/filter.h: information used to run the BPF programs themselves.
• linux/kernel/bpf/: This directory contains most of BPF-related code. In particular, those files are worth of interest:
  • syscall.c: different operations permitted by the system call, such as program loading or map management.
  • core.c: BPF interpreter.
  • verifier.c: BPF verifier.
• linux/net/core/filter.c: functions and eBPF helpers related to networking (TC, XDP etc.); also contains the code to migrate cBPF bytecode to eBPF (all cBPF programs are translated to eBPF in recent kernels).
• linux/kernel/trace/bpf_trace.c. functions and eBPF helpers related to tracing and monitoring (kprobes, tracepoints, etc.).
• The JIT compilers are under the directory of their respective architectures, such as file linux/arch/x86/net/bpf_jit_comp.c for x86. Exception is made for JIT compilers used for hardware offload, sitting in their drivers, such as linux/drivers/net/ethernet/netronome/nfp/bpf/jit.c for Netronome NFP.
• linux/net/sched/, and in particular in files act_bpf.c (action) and cls_bpf.c (filter): code related to BPF actions and filters with TC.
• linux/kernel/seccomp.c: code related to seccomp.
• linux/net/core/dev.c contains the function dev_change_xdp_fd() that is called through a Netlink command to hook a XDP program to a device, after is has been loaded into the kernel from user space. This function in turns uses a callback from the relevant driver.

Tools and utilities

bcc

• bcc framework and set of tools - One way to handle BPF programs, in particular for tracing and monitoring. Also includes some utilities that may help inspect maps or programs on the system.
• P4 compiler for BPF targets for bcc - An alternative to the restricted C.
• Lua front-end for bcc - Another alternative to C, and even to most of the Python code used in bcc.

iproute2

• iproute2 - Package containing tools for network management on Linux. In particular, it contains tc, used to manage eBPF filters and actions, and ip, used to manage XDP programs. Most of the code related to BPF is in lib/bpf.c.
• iproute2-next - The development tree, synchronised with net-next.

LLVM

• LLVM package contains several tools used in eBPF workflow. Snapshots of the latest versions for Ubuntu/Debian can be retrieved from here.
  • clang is used to compile C to eBPF object file under the ELF format (clang v3.7.1+). The BPF backend was added with this commit.
  • llvm-objdump is used to dump the content of an object file in human-readable format, possibly with the initial C source code (llvm-objdump v4.0+).
  • llvm-mc is used to compile from LLVM intermediate representation to eBPF object file, so that one can compile from C to eBPF assembly, tinker with assembly, then compile to ELF file.

bpftool and others from the kernel tree

• bpftool and other tools in the kernel tree, under linux/tools/net/ for versions earlier than 4.15, or linux/tools/bpf/ after that:
  • bpftool - A generic utility that can be used to interact with eBPF programs and maps from userspace, for example to show, dump, load, disassemble, pin programs, or to show, create, pin, update, delete maps, or to attach and detach programs to cgroups.
  • bpf_asm - A minimal cBPF assembler.
  • bpf_dbg - A small debugger for cBPF programs.
  • bpf_jit_disasm - A disassembler for both BPF flavors and could be highly useful for JIT debugging.

User space eBPF

• uBPF - Written in C. Contains an interpreter, a JIT compiler for x86_64 architecture, an assembler and a disassembler.
• A generic implementation - With support for FreeBSD kernel, FreeBSD user space, Linux kernel, Linux user space and MacOSX user space. Used for the BPF extension module for VALE switch.
• rbpf - Written in Rust. Interpreter for Linux, MacOSX and Windows, and JIT-compiler for x86_64 under Linux.

Testing in virtual environments

• A Vagrant setup - To easily test XDP. Less useful now that generic XDP (driver-independant, mostly for testing) exists.
• bcc in a Docker container.

Development and Community

• The bpf-next tree - BPF patches land in this tree. It is regularly merged into net-next, which is itself merged for each release to Linus' tree.
• Kernel documentation about contributions to BPF.
• The netdev mailing list - Mailing list for Linux kernel networking stack development. All patches are sent there for review and inclusion.
• XDP-newbies - A mailing list specially dedicated to XDP programming (both for architecture or for asking for help).
• IO Visor mailing list - BPF is at the heart of the project, and is regularly discussed on the mailing list.
• @IOVisor Twitter account.

Other lists of resources on eBPF

• IO Visor's bcc documentation
• IO Visor's bpf-docs repository
• Dive into BPF: A List of Reading Material

Acknowledgement

Thank you to Quentin Monnet and Daniel Borkmann for their original work on Dive into BPF: A List of Reading Material which became the basis for this list.

Contribute

Contributions welcome! Read the contribution guidelines first.

License

To the extent possible under law, zoidbergwill has waived all copyright and related or neighboring rights to this work.